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Foreword 


This  research  and  development  paper  is  part  of  a  continuing  effort  at  the  Navy  Personnel 
Research  and  Development  Center  (NAVPERSRANDCEN)  to  provide  information  developed 
under  research  programs  in  a  form  useable  by  Navy  personnel.  It  was  completed  under 
Independent  Exploratory  Development  Work  Unit  0602936N.RV36I27. 14  sponsored  by  the 
Technical  Director  of  NAVPERSRANDCEN  with  funding  from  the  Chief  of  Naval  Research. 

The  objective  of  this  work  unit  was  to  review  and  synthesize  evidence  from  research  on 
instruction  and  learning,  to  derive  recommendations  that  may  be  appropriate  to  guide  instructional 
practice  in  the  Navy  training  system,  and  provide  a  document  to  be  used  as  a  resource  for  managers 
of  training,  for  instructors,  and  for  those  who  design  and  develop  training. 

The  first  document  resulting  from  this  work  was  published  as  NAVEDTRA  115-1  in 
September  1988.  This  version  of  the  report  was  written  during  the  last  quarter  of  1992,  and  was 
published  as  a  Special  Issue  of  the  International  Journal  of  Educational  Research.  It  is  being 
reprinted  by  NAVPERSRANDCEN  for  wider  military  distribution. 


JOHN  D.  McAFEE 
Captain,  U.S.  Navy 
Commanding  Officer 


RICHARD  C.  SORENSON 
Technical  Director  (Acting) 
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PREFACE 


RICHARD  C.  SORENSON*  and  HERBERT  J.  WALBERG* 


'  Navy  Personnel  Research  and  Development  Center.  San  Diego.  CA.  U  S  A. 
t  University  of  Illinois  at  Chicago.  IL.  U  S. A. 


This  summary  of  research  findings  about  instruction  and  learning,  provides  practical  help 
to  individuals  serving  different  roles  in  training  and  education.  It  represents  a  synthesis 
of  the  best  available  recommendations  about  instruction. 

The  training  enterprise  is  large  and  costly.  The  investment  of  resources  for  training 
adults  worldwide  is  significant.  In  the  U.S.  alone  corporations  spend  more  than  $45 
billion  a  year  on  formal  training  programs  ranging  from  basic  skills  training  to 
management  training.  This  represents  a  50%  increase  in  five  years  (Tompson, 
1990).  In  addition.  Corporations  spend  a  great  deal  on  informal  training  such  as 
"On-the-Job-Training."  Billions  more  are  being  spent  in  military  education  and  training 
where  the  need  for  highly  trained  personnel  continues  to  increase  in  spite  of  reductions 
in  the  size  of  the  Services.  Complex  technology  requires  highly  trained  personnel. 

Over  20  million  individuals  participate  in  adult  education  programs  in  the  U.S. 
A  General  Accounting  Office  study  of  U.S.  federal  government  training  programs 
indicates  that  the  government  spends  $500  million  each  year  to  train  approximately 
one  in  three  employees.  It  is  estimated  that  individual  employees  will  need  retraining 
five  or  more  times  during  their  "working  lives."  It  is  apparent  that  the  task  of  developing 
and  maintaining  the  instructional  system  is  a  large  undertaking.  Since  we  invest  a 
large  percentage  of  our  Gross  National  Product  in  education  and  training,  even  minor 
improvements  in  instruction  can  result  in  substantial  savings.  The  intent  of  the  current 
effort  is  to  improve  instruction  by  making  information  about  effective  instructional 
practices  available  for  use. 

This  document  is  intended  to  serve  as  a  resource  for  (1)  training  managers/ 
administrators.  (2)  instructional  developers  or  designers,  and  (3)  instructors  who 
arc  involved  in  adult  education.  The  research  summary  statements  have  been 
grouped  together  in  logical  categories:  Analysis.  Design.  Development,  Presentation 
of  instruction.  Exploiting  instructional  technology.  Evaluation  and  management  of 
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instruction.  As  a  result,  the  entire  set  of  findings  about  adult  training  will  not  be  ol 
equal  interest  to  everyone:  the  projected  relevant  topics  for  each  user  population  are 
suggested  by  the  matrix  which  follows  the  Introduction. 

Over  the  last  decade  or  so  thousands  of  research  studies  have  attempted  to  tease 
out  which  instructional  techniques  are  most  effective.  Few  people  have  the  time  to 
become  familiar  with  the  resulting  literature.  Therefore,  attempts  have  been  made  to 
summarize  the  results  and  synthesize  recommendations  to  guide  instructional  practices. 
This  document  has  that  intent. 

As  might  be  expected,  some  instructional  techniques  (variables)  have  been  found  to 
have  a  greater  impact  on  student  learning  than  others.  Th'c  following  list  is  derived 
from  a  summary  by  Walberg  ( 1984)  and  by  Bloom  (1984).  where  many  research  studies 
were  reviewed  and  combined  to  provide  estimates  of  the  size  of  the  effect  of  different 
instructional  strategies.  Further,  they  are  grouped  according  to  whether  the  change  is 
directed  at  the  learner,  teacher,  materials,  or  home  and  peers.  The  size  of  the  effect 
for  each  variable  is  shown  in  standard  deviation  units.  It  is  the  average  effect  size  found 
in  a  number  of  comparisons.  With  an  effect  size  of  1.0  units,  the  average  student  in 
a  treatment  group  exceeded  about  84%  of  students  in  the  conventional  instruction 
comparison  group. 


Tabic  1 


Class 

Variable 

Effect  size 

Percentile 

equivalent 

Learner 

Corrective  feedback,  musters  learning 

!.<X> 

84 

Student  time  on  task 

1.(10 

84 

Student  participation 

1.00 

84 

Improved  readme/studv 

1.00 

84 

Materials 

New  curricula 

0.30 

82 

Advanced  organizers 

0.20 

58 

Home.  Peer 

Cooperative  learning 

0.80 

79 

Intervention  at  home 

0.50 

89 

Peer  group  influence 

0.20 

58 

Teacher 

Tutorial  msiruciion 

2.00 

98 

Reinforcement 

1.20 

88 

Cues  and  explanations 

1.00 

84 

Graded  homework 

0.80 

79 

Peer  tutorine 

0.40 

88 

Assigning  homework 

0.30 

82 

Higher  order  questioning 

0  30 

82 

Lor  contrast 

Socio-economic  class 

0.25 

80 

The  research  on  learning  and  instruction  has  things  to  teach  us  about  how  students 
learn  and  therefore,  about  the  organization  of  instruction  and  the  nature  of  tests 
that  would  facilitate  learning.  The  studies  attempting  to  identify  effective  instructional 
variables  led  to  the  development  of  summaries  describing  “What  works",  intended  to 
provide  information  for  people  broadly  concerned  with  the  organization  of  instruction. 
These  summaries  are  intended  as  advice  for  implementing  "good"  practices  in  teaching, 
as  well  as  summarizing  research.  For  example,  in  198b.  Secretary  of  Education  W.  J. 
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What  Works  in  Adult  Instruction 


Bennett  published:  What  works:  Research  about  teaching  and  learning  directed  primarily 
at  parents  and  teachers  of  young  children.  A  similar  document:  What  works:  Summary 
of  research  findings  with  implications  for  Navy  instruction  and  learning  soon  followed, 
but  focused  on  the  needs  of  administrators,  instructors  and  educational  specialists  who 
instruct  Navy  recruits  (Montague.  1988).  That  publication  was  the  result  of  surveying 
the  extensive  relevant  research  and  then  having  the  conclusions  evaluated  or  judged  by 
a  panel  consisting  of  W.  E.  Montague  (NPRDC),  F.  G.  Knirk  (University  of  Southern 
California).  J.  A.  Kutik  (University  of  Michigan).  M.  D.  Merrill  (Utah  State  University). 
.1.  Orlansky  (Institute  for  Defense  Analyses).  E.  Rothkopf  (Columbia  University).  T. 
Sticht  (Applied  Behavioral  &  Cognitive  Sciences),  and  H.  Walberg  (University  of 
Illinois.  Chicago). 

The  current  document  focuses  on  adults,  and  is  an  extension  and  revision  of  the  Navy 
document  for  a  broader  non-Navy  audience  of  educators  and  trainers  who  are  involved 
with  adult  education.  It  is  addressed  to  educators  involved  in  the  delivery,  management, 
planning,  design,  and  development  of  education  and  training. 

A  basic  assumption  is  that  instructional  content  and  methodology  decisions  make  a 
difference,  and  that  it  is  desirable  to  reduce  training  costs  and  improve  effectiveness. 
Thus,  individuals  involved  in  training  administration,  design  or  instruction,  can  benefit 
from  concise  information  about  what  research  says  makes  a  difference.  What  Works. 
Recommendations  have  been  made  based  on  the  robustness  of  support  from  the  research 
literature  and  the  likelihood  that  changes  can  be  made  in  teaching.  They  are  stated  in 
an  easily  comprehensible  form  to  guide  instructional  practice,  evaluation,  planning,  and 
policy  making. 

The  information  in  this  summary  is  a  distillation  of  experience  (panel  of  experts) 
and  the  research  in  education,  military  training,  vocational  education,  and  business 
training.  We  hope  it  is  a  useful  distillation.  We  hope  that  instructors,  training 
executives/administrators,  instructional  developers,  and  supervisors  are  willing  and 
able  to  improve  their  own  schools  and  education  in  general. 


References 

Blouin.  B  (1984).  The  2-sigma  problem:  The  search  for  methods  of  group  instruction  as  effective  as 
one-to-one  tutoring  Educational  Researcher.  13,  4-16. 

Montague.  W.  E  (Ed.)  ( 1  OSS ) .  What  works:  Summary  of  research  findings  with  implications  for  Saw 
instruction  and  learning.  NAVEDTRA  115-1.  Pensacola.  FL:  Chief  of  Naval  Education  and  Training. 
Tompson.  B.  (1990)  Training  salary  survey.  Training. 

Walberg.  H  J  (1984).  Improving  the  productivity  of  America's  schools.  Educational  Leadership.  41(8). 
19-27. 
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INTRODUCTION 


WILLIAM  E.  MONTAGUE*  and  FREDERICK  G.  KNIRKt 


Overview  of  the  Special  Issue 

The  basic  assumption  behind  this  book  is  that  current  research  on  instruction,  student 
learning,  and  performance  has  important  things  to  tell  us  about  how  students  learn  and 
perform.  Therefore,  this  knowledge  can  help  with  planning  and  organizing  instruction, 
testing  learning  performance,  and  managing  instruction  efficiently  and  effectively.  Often, 
those  involved  in  planning  instruction  are  unfamiliar  with  results  from  research  that  can 
guide  instructional  practices.  These  results  are  scattered  through  many  journals  that  are 
not  readily  available,  and  articles  often  report  results  from  individual  studies  that  may 
be  hard  to  interpret  and  are  of  unknown  reliability. 

The  purpose  of  this  special  issue  is  to  provide  summaries  of  research  results  for 
people  who  must  design  and  deliver  instruction.  In  the  international  industrial  and 
military  training  community  these  individuals  are  mostly  subject  matter  specialists  not 
extensively  versed  in  educational  research.  Thus,  it  is  important  for  them  to  have  sources 
of  understandable,  accurate  and  reliable  information  about  “what  works"  in  educating 
and  training  young  adults.  The  basic  assumption  we  make  is  that  this  information  can 
be  useful  for  instructional  decisions  and  improve  instruction.  Individuals  involved  in 
instructional  administration,  design,  or  delivery  can  benefit  from  concise  descriptions 
of  factors  that  consistently  improve  student  learning:  What  works.  This  document  is 
meant  to  provide  recommendations  in  an  easily  comprehensible  form  as  a  guide  for 
instructional  practice,  evaluation,  to  help  planning  and  policy  making. 

This  summary  of  research  findings  began  as  a  project  to  provide  recommendations  to 
the  U.S.  Navy's  education  and  training  system  to  improve  schooling,  instructor  training, 
and  manager's  training.  Information  was  distilled  from  research  and  practice  specifically 
geared  to  education  and  training  in  Navy  schools.  A  group  of  expert  advisors  was  used 
to  assist  in  selecting  the  entries  to  be  useful,  and  important  for  Navy  education  and 
training.  Some  of  these  advisors  had  been  involved  in  analyzing  and  synthesizing  research 
evidence,  and  others  were  involved  in  administering  and  managing  schooling. 

We  examined  how  education  and  training  was  accomplished  in  the  Navy,  and  identified 
three  broad  groups  of  users  who  play  different  roles  in  the  training  system.  We  examined 
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policies  and  practices  to  identify  constraints  under  which  the  groups  work.  Then,  we 
collected  and  re\  iewed  the  research  literature  to  identify  material  relevant  for  the  groups 
Entries  were  included  only  when  research  evidence  and  expert  opinion  suggested  thev 
were  stable  and  consistent.  The  difficult  part  of  the  effort  was  to  translate  the  research 
findings  into  clear  and  comprehensible  statements  that  we  think  can  be  used  In  users 
to  guide  their  practices. 

We  developed  a  book.  W'hai  works:  Summary  of  research  findings  with  implications  for 
V <n  v  instruction  and  learning  (Montague.  1988),  providing  some  of  the  most  up-to-date 
reference  information  for  the  three  broad  classes  of  personnel  that  run  Navv  schooling 
There  are  Training  Executives  who  manage,  administer,  and  supervise  schooling  at 
various  levels.  Instructors  are  mostly  military  personnel  who  are  assigned  as  part  of  their 
job-rotation.  They  are  briefly  trained  to  be  teachers,  and  are  assigned  as  instructors  for 
short  periods  (up  to  three  years).  Training  Specialists  are  the  third  group.  Thev  provide 
educational  advice  and  assistance  to  the  other  groups.  They  help  with  evaluation,  course 
materials  design  and  development.  In  general,  these  groups  may  have  little  formal 
training  about  instruction,  instruction  development,  or  management  of  instruction, 
although  Training  Specialists  are  likely  to  have  some.  The  same  classes  of  personnel  are 
generally  involved  with  teaching  and  instructional  design  and  development  in  industrial 
and  educational  organizations. 

The  preparation  of  the  Navy's  book  was  motivated  by  a  similar  document  produced  by 
the  Department  of  Education.  That  document.  What  works:  Research  about  teaching  and 
learning  (Bennett,  1986),  became  the  most  widely  distributed  document  on  instructional 
research  yet  published.  It  was  directed  primarily  at  teachers  and  paret  ts  of  young 
children  attending  public  schools  in  the  United  States.  In  the  Navy  document  and 
in  the  current  volume,  our  primary  concerns  are  the  instruction  of  adults  choosing 
practical  careers  rather  than  children. 

Each  '•finding''  was  chosen  in  terms  of  the  robustness  of  the  research  support,  and 
its  appropriateness  for  one  or  more  of  the  groups  involved  in  schooling.  The  references 
cited  with  each  finding  should  be  viewed  as  representative  of  the  research  evidence  or 
for  providing  information  on  implementation  methods.  Thus,  the  information  in  this 
volume  is  a  distillation  of  experience  and  a  large  body  of  scholarly  research  in  education, 
military,  and  vocational  education  and  training.  It  consists  of  discrete  findings  about 
teaching  and  learning  that  may  be  broadly  applicabk  in  focused,  adult  education  and 
training  classrooms,  for  instructional  development  and  in  planning  how  instruction  will 
be  done  in  the  future.  In  some  cases  the  findings  may  simply  support  what  is  current 
practice.  In  others,  it  may  provide  guidance  to  dramatically  change  current  instruction 
or  planning  for  future  instruction. 

The  findings  are  organized  into  categories  that  correspond  roughly  to  stages  in  the 
instructional  design  and  development  process.  The  use  of  systematic  design  procedures 
can  make  instruction  more  effective  and  efficient  than  it  traditionally  has  been.  A 
large-scale  test  of  the  effectiveness  of  instructional  system  design  methods  was  its  use 
to  revise  the  elementary  and  middle  school  education  system  in  the  Republic  of  Korea. 
Students  in  the  redesigned  groups  outperformed  those  in  control  groups  in  all  school 
subjects  by  about  2(1%  on  average  (Morgan.  1989). 

At  the  least,  all  models  of  instructional  design  (and  there  arc  many.  Gustafson. 
1981)  call  for  analyzing  what  is  to  be  learned,  specifying  who  is  to  learn,  devising 
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instructional  methods,  and  designing  evaluation  procedures  to  determine  effectiveness 
and  to  provide  checks  on  the  process.  The  design  models  describe  the  process  tor 
designing  and  developing  instruction.  The  intent  is  to  allow  designers  to  determine 
the  best  ways  to  implement  the  process.  The  process  has  considerable  flexibility,  and 
choices  m  instruction  and  evaluation  are  left  to  the  skill  and  knowledge  of  the  designer. 
Thus,  the  beliefs  and  knowledge  (or  lack)  of  designers  and  developers  determine  the 
wav  content  is  specified,  the  way  instruction  and  evaluation  of  learning  is  accomplished 
Since  this  knowledge  should  depend  on  evidence  and  theory  from  research,  our  intent 
is  to  provide  research-based  findings  and  recommendations  to  help  with  the  design 
choices  made. 


Research  Findings  Seem  Obvious 

Main  people  find  findings  from  soc'  d.  psychological,  and  educational  research 
"obvious"  because  they  seem  familiar  a  id  to  be  "just  common  sense"  knowledge.  But. 
what  is  called  common  sense  is  not  common  practice.  An  examination  of  schoolhouse 
practices  reveals  that  the  findings  are  not  known,  or  go  unheeded.  For  example.  Latham 
( 19NK)  concluded  that  training  practitioners  largely  ignore  results  from  the  research 
literature.  If  the  goal  is  to  bring  about  application  of  practices  that  have  been  shown 
to  improve  student  achievement,  and  ultimately,  their  job  proficiency,  some  means 
must  be  found  to  overcome  such  ignorance  and  inertia.  The  fact  is  that  the  feeling  that 
the  recommendations  arc  obvious  is  a  common  reaction.  It  promotes  ignoring  what  is 
known  in  favor  of  convenience  or  habit. 

In  a  recent  paper  Gage  (1991)  pointed  out  that  the  feeling  that  a  research  result 
is  obvious  common  sense  is  untrustworthy.  He  described  experiments  where  people 
were  given  statements  to  rate  as  obvious  or  unobvious.  Each  item  either  stated  an 
actual  finding  on  teaching,  or  the  opposite  of  the  true  finding.  Both  true  and  false 
statements  were  rated  obvious.  Gage  concluded  that  "People  tend  to  regard  as  obvious 
almost  any  reasonable  statement  made  about  human  behavior"  (Gage.  1991.  p.  15). 
Therefore,  we  suggest  that  generalizations  from  the  research  literature  should  not  be 
dismissed  as  "trivial"  or  "known"  information.  Instructional  practice  is  often  contrary 
to  recommendations  derived  from  the  findings,  and  improvements  are  needed  to  help 
increase  instructional  effectiveness.  For  example,  the  benefits  of  spaced  practice  have 
been  known  to  researchers  for  100  years,  yet  spaced  repetition  is  not  a  major  component 
of  instructional  programs  (Druckman  &  Bjork,  1991).  Properly  applied,  the  information 
in  this  volume  vvill  assist  the  process  but  may  have  to  overcome  resistance. 

The  earlier  versions  of  What  Works  have  been  evaluated  by  examining  their  usefulness 
for  user  groups.  The  concluding  article  in  this  issue,  by  Robertson  and  Whitehill. 
indicates  the  degree  to  which  What  Works  conclusions  have  been  used,  and  are  trusted, 
by  its  users.  It  is  interesting  to  note  that  respondents  reported  that  they  were  already 
familiar  with  the  findings.  This  observation  is  congruent  with  Gage's  result.  We  doubt 
that  this  "feeling  of  knowing"  truly  represents  the  state  of  their  knowledge. 
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Current  Perspectives  on  Learning  and  instruction 

An  understanding  of  learning  theory  is  fundamental  to  instructional  design  and 
development.  It  influences  objectives  specification,  testing,  curriculum  materials, 
evaluation  and  validitv  judgements.  Managers,  instructional  specialists,  instructors  and 
developers  all  use  a  svstems  approach  that  encourages  the  close  alignment  of  objectives, 
tests  and  curriculum  materials.  The  learning  theory  which  was  embedded  in  the  systems 
development  approach  when  it  was  first  applied  to  instructional  development  derives 
from  behavioristic  learning  theory  which  requires  the  sequential  mastery  of  constituent 
skills  and  behaviorally  explicit  testing  of  each  step  in  learning.  This  model  of  "learn  the 
sequential  .acts  before  understanding"  has  been  contradicted  by  a  substantial  body  of 
recent  research  in  cognitive  psychology  (Shepard.  1991). 

Several  problems  result  from  using  this  model  to  develop  instruction.  First,  it  leads  to 
decontextualization  of  objectives,  tests  and  material.  The  goal  is  to  learn  the  declarativ  e 
knowledge  (facts,  formal  concepts  etc.)  of  a  particular  subject  matter  (e.g..  physics 
theory  background  for  electronics),  or  general  computational  skills  (e.g..  algebra),  rather 
than  as  information  to  be  used  (tool-like)  to  do  practical  tasks  (Bransford.  Sherwood. 
Husselbring.  Kinzer.  &.  Williams.  1990;  Brown.  Collins.  &  Duguid.  1989).  Secondly,  the 
model  assumes  that  component  skills  can  be  defined  adequately,  learned  independent^ . 
and  out  of  context.  More  advanced  skills  are  learned  by  assembling  component  skills. 
This  leads  to  delaying  teaching  higher  order  skills  until  after  prerequisite  skills  have 
been  mastered  (Shepard.  1991 ).  Also,  this  perspective  leads  to  testing  facts  using  various 
forms  of  objective  tests  (e.g..  multiple-choice)  without  evidence  for  the  correspondence 
between  the  cognitive  processes  involved  in  the  test  and  the  criterion  performance,  in 
addition,  student's  perspectives  about  learning  goals  are  influenced  by  them  being  asked 
simple  questions  requiring  recall  or  recognition  of  facts  or  procedures  out  of  context.  Seldom 
are  they  asked  to  describe  how  they  would  approach  performing  the  task,  which  would  be 
a  preferred  w  ay  of  testing  their  practical  knowledge. 

Contemporary  cognitive  psychological  research  suggests  that  this  model  is 
inadequate.  Studies  find  that  things  are  easier  to  learn  if  they  are  learned  in  a 
context  of  use.  when  they  make  sense  to  the  student.  Learning  is  characterized 
by  students  taking  in  information,  interpreting  it.  relating  it  to  what  they  know 
already  and  when  necessary,  reorganizing  their  cognitive  structures  to  accommodate 
new  understanding.  Students  construct  and  reconstruct  mental  models  or  schemata 
that  organize  their  ideas  and  inter-relationships.  The  situation  in  which  knowledge 
is  developed  and  used  is  not  ancillary  to  learning  or  separable  from  it.  It  is  an 
integral  part  of  learning  and  cognition  (Brown  el  a!.,  1989).  Thus,  we  recognize  that 
learning  requires  a  high  level  of  activity  from  the  learner.  This  needs  to  be  supported 
by  good  teaching.  Situational  representations  need  to  be  devised  that  provide  the 
context  for  use.  K.  Glaser  portrayed  how  instruction  should  be  organized  to  ensure 
learner  activity: 

"When  schema  knowledge  is  viewed  as  a  set  of  theories,  it  becomes  a  prime  target  for 
instruction.  We  can  view  a  schema  as  a  pedagogical  mental  structure,  one  that  enables 
learning  bv  facilitating  manors  retrieval  and  the  learner's  capacity  to  make  inferences 
on  the  basis  of  current  knowledge.  When  dealing  with  individuals  who  lack  adequate 
knowledge  organization,  wc  must  provide  a  beginning  knowledge  structure.  This  might  he 
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have  called  them,  are  reeularh  deeised  be  ingenious  teachers  Such  structures  when  the-, 
are  interrogated,  instantiated,  or  lalsttied.  help  organize  ness  know ledge  and  offer  a  basis 
tor  problem  solving  that  leads  to  the  formation  of  more  complete  and  expert  schemata  The 
process  ot  knowledge  acquisition  can  be  seen  as  the  successive  development  of  structures 
which  arc  tested  and  modified  or  replaced  in  wavs  that  facilitate  learning  and  thinking 
( C  I  laser  IdM  p  in  1  i 

The  aruiKsis  of  tasks  to  be  learned  needs  to  reflect  these  ideas  and  the  ideas  are  often 
at  odds  with  traditional  ways  of  objective  specification  and  designing  tests  to  measure 
progress  and  facilitate  learning. 


"Tests  ouithf  not  ask  tor  demonstration  ot  small  discrete  skills  practiced  in  isolation  The\ 
should  be  more  ambitious  instruments  aimed  at  detecting  what  mental  representations 
students  hold  ot  important  ideas  and  what  tacihtv  students  have  in  bringing  these 
understandings  to  bear  in  solving  new  problems  "  (Shepard.  |d*Jf.  p  d  ) 

The  goal  of  instructional  design  and  development  is  to  engineer  conceptual  growth 
and  understanding.  Instruction  should  center  on  the  student's  attempt  to  construct 
understanding.  It  should  be  devised  to  support  the  student's  attempts  to  interpret, 
structure  or  restructure,  and  use  new  knowledge,  as  opposed  to  the  memorization  and 
verbatim  reproduction  of  information.  Instructional  environments  that  foster  learning, 
encourage  questioning,  evaluating  one's  own  and  others'  performance,  and  critiquing 
the  performance  process. 

Objecmos  and  tests  ought  to  have  the  goal  of  capturing  in  the  test  tasks  the  same 
requirements  for  cognition  and  knowledge  integration  as  required  in  the  criterion 
performance  desired.  Tests  serve  to  assess  a  student's  understanding,  but  also  are 
important  instructional  tools  To  convey  to  students  that  their  reasoning  is  more 
important  than  choosing  a  correct  answer,  we  might  ask  them  to  show  how  to  set 
up  an  approach  to  a  task,  to  explain  a  point  to  another  student,  or  to  plan  a  route 
to  get  from  point  A  to  point  B.  This  can  be  thought  of  as  authentic  testing. 

The  notion  of  using  authentic,  direct  performance  assessments  in  testing  is  repeated 
in  recommendations  for  instruction.  Instruction  should  be  centered  around  authentic 
task  practice  situated  in  a  context  of  use  (Gott.  1989).  Students  are  to  be  taken  through 
successive  approximations  of  mature  practice  while  learning  to  perform  specific  tasks. 
The  method  is  referred  to  with  several  names,  "cognitive  apprenticeship"  comes  closest 
to  being  descriptive  (Brown  et  al..  1989).  Cognitive  apprenticeship  supports  learning 
in  a  domain  by  providing  authentic  domain  activity,  allowing  students  to  experience 
the  full  functional  context.  The  student  learns,  develops  and  uses  the  cognitive  tools 
appropriate  for  the  domain.  It  is  similar  to.  but  less  resource  demanding  than,  traditional 
craft  apprenticeship  since  it  is  intended  to  be  useful  in  schooling. 

The  term  "apprenticeship"  emphasizes  the  importance  of  “...activity  in  learning  and 
knowledge  and  highlights  the  inherently  context-dependent,  situated  and  enculturating 
nature  of  learning"  (Brown  et  al..  1989.  p.  39).  It  also  suggests  that  to  promote  student 
learning,  instructors  must  make  their  own  tacit  knowledge  explicit,  and  model  how 
to  perform  the  task.  Then,  the  instructor  has  the  student  perform  independently  and 
offers  hints,  feedback,  reminders,  or  redirects  the  student's  attention  to  salient  features. 
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The  coal  is  to  make  the  student's  performance  approximate  the  expert's  as  closely  as 
possible.  The  word  'cognitive’  means  that  apprenticeship  is  more  than  the  physical 
skills  usualh  connected  with  it  The  apprentice  must  learn  to  reason,  speak,  and  act 
like  a  practitioner. 

Research  programs  investigating  practical  skill  learning  have  thus  led  to  a  change  in 
the  understanding  of  processes  involved  in  learning  and  performance.  They  have  changed 
our  perspective  about  how  instruction  in  practical  tasks  should  be  done  and  how  s?  nt 
progress  should  be  assessed.  Rather  than  identify  task  components  and  teach  them  out 
of  context,  the  advice  is  to  teach  in  the  task  context  providing  support  (advice,  cuing)  for 
students  so  that  they  can  build  realistic  experiences.  We  have  attempted  to  incorporate 
this  perspective  into  the  recommendations  that  follow  and  to  the  summary  findings  that 
form  the  bodv  of  this  document. 


Comrnunications  Computer  Technology  can  Help  Instruction 

The  performance  orientation  of  the  prior  discussion  implies  that  instruction  and 
testing  learning  performance  will  involve  simulation-like,  authentic,  representations  to 
mediate  student  learning.  Obviously,  new  communications  and  computer  technologies 
can  be  used  for  such  simulations.  The  increasing  availability  and  declining  costs  of  this 
gugetry  and  its  popularity  will  ensure  that  it  will  be  used  in  education  and  training.  We 
are  worried  that  use  of  the  technologies  will  occur  without  careful,  systematic  effort  to 
incorporate  what  we  know  about  learning  principles  to  promote  effective  learning  and 
performance. 

The  role  of  the  designer  of  instruction  is  to  provide  the  learner  with  the  necessary 
tools  and  conditions  for  learning.  The  student  needs  to  acquire  the  appropriate  language 
and  concepts  to  be  used  to  understand  and  describe  situations  in  which  what  is  learned 
is  used.  The  student  needs  to  learn  how  to  put  the  information,  facts,  situations,  and 
performance  skill  together  in  appropriate  contexts. 

Competent  performance  of  complicated  tasks  develops  slowly,  and  requires  an 
environment  that  supports  the  development  of  subskills,  knowledge  and  their 
coordination.  Anyone  attempting  to  design  and  develop  instruction  has  a  substantial 
orchestration  problem  on  their  hands.  It  requires  analysis  of  competence  or  skill  and 
breaking  it  into  manageable  and  appropriate  chunks  for  students  to  learn.  Providing  a 
learning  environment  requires  considerable  sophistication  and  skill  for  using  resources. 
It  also  requires  understanding  the  uses  to  which  competencies  can  be  put.  An  attractive 
feature  is  the  use  of  technology  for  representing  task  situations  to  students  realistically. 
Representing  cases,  situations,  scenarios,  and  simulations,  to  approximate  actual  task 
performance  conditions  requires  technological  assistance  and  considerable  resources. 

Computer  and  modern  communications  technologies  can  help  in  various  ways  with 
the  preparation  of  instrtu  m  and  its  use.  For  example,  in  the  design  and  development 
stages  of  instructional  preparation  computer-based  tools  can  assist  with  the  analysis  of 
tasks,  and  planning  instructional  sequences.  Authoring  systems  can  aid  preparation  of 
instructional  materials  and  coordinating  related  segments  of  instruction  that  may  need 
frequent  revision. 

Technology  is  likely  to  aid  the  delivery  of  instruction  in  various  ways.  Access  to 
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instruction  can  be  increased  using  modern  communications  technology  The  "Open 
University"  idea  has  worked  well  tor  years  and  the  technology  can  help  remove  barriers 
to  education  and  training.  From  the  distribution  side,  technology  can  pros  ide  widespread 
delivery  of  presentations  at  remote  locations  at  varied  times.  Forms  of  individualized 
learning  have  proven  highly  effective  and  some  of  these  forms  can  be  implemented 
broadly  using  computer-based  and  video  technologies. 

Providing  learning  environments  that  are  functionally  equivalent  to  target 
work/application  situations  will  need  the  assistance  of  technologies,  both  from 
educational  design  as  well  as  computer/communication  technology  But.  these 
'  requirements  for  realism ”  are  costly  to  implement,  requiring  special  skills. 

The  evaluation  of  the  adequacy  of  instruction  as  well  as  student  progress  toward 
competence  can  be  aided  by  use  of  computer/communication  technology  Thus,  there 
are  mans  ways  in  which  technology  can  assist  the  instructional  process.  This  has  been 
recognized  for  many  years.  To  make  effective  and  efficient  use  of  technology  requires 
systematic  planning  and  assessment,  which  does  not  happen  often,  or  easily 

The  Office  of  Technology  Assessment  (198S)  said  "A  more  focused  effort  to  expand 
substantially  the  use  of  technology  in  education  and  attain  more  fully  integrated 
applications  across  the  curriculum  will  probably  requtre  new  strategies  ...  "  ( p.5 >. 
Due  to  the  importance  of  these  instructional  technologies  a  final  section  in  What 
works  summarizes  yvhat  vve  kmnv  about  their  use  for  adult  training. 

The  Office  of  Technology  report  ( 1988)  supports  the  use  of  instructional  technologies 
in  the  classroom.  It  says  "By  and  large,  the  research  to  date  supports  the  continued  use 
of  instructional  technologies  in  the  schools  .  .  .  there  is  evidence  that  computer-assisted 
instruction  can  raise  achievement  test  scores  for  some  students;  but  there  is  also  wide 
agreement  that  computer  technologies  can  already  do  more  than  provide  electronic 
equivalents  of  drill  and  practice  workbooks,  and  that  much  of  their  future  promise 
lies  in  experimentation  and  development  of  nontraditiona!  learning  methods"  (p.  42). 
Computer-based  systems  may  assist  in  implementing  the  authentic  instruction  discussed 
previously.  Sophisticated  uses  of  technology  such  as  this,  however,  require  considerable 
expertise  in  program  development  and  substantial  changes  in  the  instructional  system  as 
a  whole. 

Training  managers  can  use  this  technology  to  make  them  much  more  effective.  The 
Office  of  Technology  Assessment  (1988)  looked  at  these  communications  technologies 
as  follows;  "OTA  finds  that  investments  in  technology  cannot  be  fully  effective  unless 
teachers  receive  training  and  support”  (p.  16). 

"OTA  finds  th;ii  software  manufacturers  tend  to  play  it  safe.  They  produce  what  (will 
sell)  and  teachers  usually  huy  products  that  are  familiar.  The  potential  result  is  a  relatively 
homogeneous  set  of  products  that  fall  far  short  of  the  possibilities  provided  by  the  new 
learning  tools"  (p  22). 

In  an  effort  to  assist  the  different  categories  of  readers  (training  managers, 
instructional  developers,  and  classroom  instructors)  to  the  most  appropriate 
materials,  a  "What  Works:  User  Matrix”  has  been  constructed.  The  matrix  follows 
this  introduction. 

The  potential  for  technology  to  increase  the  quality,  effectiveness,  and  efficiency  of 
education  and  training  depends  on  careful  design  and  careful  implementation.  Neither 
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in  eas\.  In  spite  of  adoption  of  systems  design  procedures,  the  development  and 
management  of  instruction  is  quite  variable.  The  procedures  do  not  make  up  for  deficit'- 
in  the  '-kills  needed  to  devise  quality  instruction.  Computer-Zcommunication-aids  may 
help,  but  they  need  to  be  developed  and  made  available  broadly  to  have  an>  significant 
impact.  Their  development  will  require  significant  resources  that  are  currently  in 
short  supple.  In  addition,  although  some  gains  might  be  realized  by  implementing 
computer-based  and  other  technologies,  the  gains  are  realized  only  when  the  way 
schooling  is  done  changes.  The  way  instructors  are  utilized  and  schoolhouses  are 
managed  needs  to  change  to  accommodate  the  technologies.  It  seems  likely  that 
technology  will  gradually  be  adopted  to  help  the  process,  and  unlikely  that  it  will 
change  things  radically. 


Role  Descriptions  for  Training  Practitioners 

An  analysis  of  the  tasks  of  instructional  personnel  coupled  with  relevant  research  led 
us  to  develop  the  following  list  of  suggestions.  This  is  an  attempt  to  focus  attention  on¬ 
to  certain  aspects  of  their  roles  and  responsibilities  to  assist  in  coordinating  the  roles 
with  recommendations  from  research. 


Managers/ Administrators 

•  Become  assertive  instructional  leaders  by  putting  instructional  excellence  first. 

•  Bring  instructional  technology  and  good  practices  to  bear  on  instruction. 

•  Demand  high  quality  in  training  from  staff,  instructors  and  students. 

•  Develop  and  monitor  in-service  staff  training. 

•  Establish  a  system  for  evaluation  and  monitor  it  systematically. 

•  Promote  a  positive  climate  and  overall  atmosphere. 

•  Focus  programs  on  instructional  goals  and  protect  them  from  irrelevant  demands. 

•  Encourage  consensus  on  values  and  goals. 

•  Plan  and  coordinate  long-range  changes  in  training  to  increase  effectiveness  and 
efficiency. 

•  Analyze  and  plan  for  use  of  instructional  technologies  to  increase  training 
productivity  due  to  informed  media,  and  class  size  decisions. 

•  Consult  with  training  specialists  about  training  policy  and  practices. 

•  Provide  human  factors  controls  in  classrooms  so  instructors  can  direct  attention  and 
learning  via  light,  color,  temperature,  and  noise  control. 


Instructional  Developers 

•  Learn  and  understand  scientific  bases  for  training  excellence. 

•  Become  assertive  instructional  leaders  by  emphasizing  factors  that  bring  about 
excellence. 

•  Expect  high  quality  and  productivity  from  staff,  instructors,  and  students. 
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•  Implement  and  monitor  in-service  staff  training. 

•  Monitor  and  evaluate  instructors,  and  instruction. 

•  Promote  peer  interaction  among  instiuctors. 

•  Protect  instruction  from  irrelevant  demands. 

•  Develop  a  well-structured,  work-like  training  environment  to  support  student 
learning. 

•  Adjust  training  to  goals  and  to  learners  through  detailed  evaluation  of  performance. 

•  Assist  instructors  in  providing  feedback  to  students. 

•  Monitor  development  and  empirical  evaluation  of  training  technologies. 

•  Analyze  and  propose  improvements  in  training  effectiveness  and  efficiency. 

•  Provide  input  to  higher  management  regarding  training  policy. 


Classroom  Instructors 

Bring  good  practices  to  bear  on  training. 

Focus  classroom  activities  on  learning. 

Emphasize  student  learning  and  achievement. 

Monitor  student  studying  and  adjust  their  activities  to  maximize  their  effort  and 
progress. 

Promote  effective  use  of  instructional  time  in  learning. 

Learn  teaching  techniques  that  enhance  student  learning. 

Provide  well-structured  presentations  and  classroom  activities. 

Arrange  many  and  varied  learning  opportunities. 

Create  a  job-like  instructional  situation. 

Emphasize  hands-on.  job-like  performance  tests. 

Use  testing  and  questioning  to  evaluate  learning  progress  and  maintain  motivation 
to  learn. 

Give  corrective  feedback  regularly. 

Provide  students  with  opportunities  for  individualized  work. 

Design  out-of-class  assignments  to  increase  student  achievement. 


Instructional  Personnel  can  Exploit  Instructional  Hardware  and  Software 

•  Learn  the  basis  for  ergonomic  principles  and  guidelines  involved  in  designing 
educational  equipment  and  furnishings. 

•  Become  assertive  instructional  leaders  by  not  settling  for  communications  equipment 
developed  for  non-educational  uses,  and  require  communications  equipment  which 
facilitates  good  learning  and  perception  principle  implementation. 

•  Expect  high  technical  and  design  quality  in  all  courseware  (films,  video  programs, 
computer  software,  etc.). 

•  Monitor  development  and  empirical  evaluation  of  training  hardware  and  software 
technologies. 
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Summary  Statements 

In  the  main  body  of  this  document,  "findings"  and  modifying  "comments"  about 
each  of  the  what  works  topics  are  presented.  A  finding  presents  a  generality  about 
an  instructional  practice  variable  that  falls  in  one  of  the  stages  of  instructional  design. 
The  comments  provide  explanation  and  reference  information  about  each  finding.  The 
items  represent  heuristic  guidance  to  assist  designing  instruction,  and  can  also  be  used 
for  evaluating  practices  to  detect  deficiencies.  We  are  not  sure  about  the  effects  of 
combining  effective  instructional  variables.  Presumably,  the  combination  will  increase 
the  benefit  for  student  learning.  But.  some  combinations  will  likely  have  uncertain,  or 
even  negative,  effects  on  student  learning.  Skill  in  using  this  sort  of  information  is  sure 
to  be  developed  through  experience.  Thus,  the  findings  provide  general  statements 
about  what  factors  have  been  found  to  affect  student  learning.  Their  use  in  practice, 
will  be  determined  by  constraints  on  resources  and  people. 
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Table  2 

What  Works:  User  Matrix 


Managers  Designers  Instructors 


Analysts 


1.  Training  objectives 

M 

H 

H 

2.  Training  analysis  provides  course  objectives 

M 

H 

M 

Design  and  development 

3.  Focus  on  post-training  performance 

M 

H 

H 

4  Refresher  training 

M 

M 

H 

5.  Increasing  time  on  task 

M 

H 

H 

6.  Distributing  practice  in  time 

M 

H 

H 

~.  Development  of  mental  models 

L 

H 

M 

8.  Learning  is  based  on  prior  knowledge 

L 

H 

M 

9.  Motivating  students 

L 

H 

H 

10.  Cooperation  in  learning 

L 

H 

M 

11  Situated  teaching  and  learning 

L 

H 

M 

12.  Student  control  of  learning 

L 

H 

H 

13.  Writing  text  materials 

L 

H 

L 

14  Readability  of  training  materials 

M 

H 

M 

15  Memorization  aids 

L 

H 

M 

16  Using  examples  and  nonexamples 

L 

H 

M 

Presentation  of  instruction 

17  Instructor  classroom  leadership 

L 

M 

H 

18  Instructor  classroom  role 

M 

H 

H 

19.  Teaching  students  how  to  learn 

L 

H 

M 

20.  Instructor  presentation  stimulates  learning 

L 

H 

H 

21.  Practice 

L 

H 

H 

22.  Peer  teaching 

L 

H 

H 

23  Listening  skills 

L 

H 

M 

24.  Homework  assignments 

M 

H 

H 

25.  Testing  student  learning 

H 

H 

M 

26.  Giving  feedback  to  students 

L 

H 

H 

Exploiting  instructional  technology 

27.  Cost  effectiveness 

M 

H 

L 

28.  Programming  instructional  presentation 

M 

H 

L 

29.  Programmed  instruction  materials 

L 

H 

H 

30.  Compute) -based  instruction 

M* 

H 

M 

31.  Hypertext-based  instruction 

L 

H 

L 

32.  Visual  ^presentations  for  learners: 
television/imeractive  video/ V  IT 

M* 

H 

M 

33  Simulations  and  simulation  games 

M* 

H 

M 

34.  Distributed  instruction 

L* 

H 

L 

35  Job  performance  aids 

L 

H 

M 

36.  Adopting  training  innovations 

H 

H 

L 

37  Designing  effective  illustrations  and  graphs 

L 

H 

M 

Continued 
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Table  2 
(Continued) 


Managers 

Designers 

Instructors 

Evaluation 

38.  Criterion  referenced  testing 

L 

H 

M 

39.  Course  evaluation  and  revision 

H 

H 

H 

4(1.  Evaluating  and  supervising  instructors 

H 

L 

L 

Management  of  instruction 

41.  Planning  change  within  organizations 

H 

M 

L 

42.  Manaeing  instructors 

H 

L 

L 

43  Managing  student  learning 

H 

H 

H 

44.  Monitoring  and  tailoring  instruction 

L 

H 

M 

45.  Phvsical  classroom  environment 

M 

H 

M 

46.  Manaeing  informal  learning 

L 

M 

M 

47.  Student-instructor  ratio  tradeoffs 

H 

H 

M 

H  —  This  topic  will  be  of  high  interest  to  this  audience:  M  —  This  topic  will  be  of  medium  interest 
to  this  audience:  L  —  This  topic  will  be  of  low  interest  to  this  audience;  '  —  These  topics  need  to  be 
considered  by  course  managers  because  of  the  high  initial  costs  for  hardware  acquisition  and/or  software 
development. 
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FINDINGS  MATRIX: 

WHAT  WORKS  IN  ADULT  TRAINING 


FINDINGS  ABOUT  THE  ANALYSIS  OF  INSTRUCTION 

The  design  of  instruction  can  be  viewed  as  a  problem-solving  process,  a  way 
of  solving  instructional  problems.  The  first  steps  in  the  process  are  analytical 
Determining  whether  training/instruction  is  needed:  analyzing  what  must  be 
learned,  by  whom,  and  specifying  the  goals  and  objectives  that  serve  as  the 
focus  for  organizing  instruction  and  evaluation. 


345 
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1.  Training  Objectives 


Finding:  Instructional  objectives  reflecting  the  instructional  requirements 

directly  guide  instruction  and  testing  and  thus  student  learning. 


Comments:  The  use  of  measurable  or  observable  instructional  objectives  helps 

ensure  consistency  between  what  is  learned,  course  content,  and  test 
items.  This  consistency  or  congruity  reduces  the  amount  of  irrelevant 
course  material,  and  objectives  can  assist  student  learning. 

When  materials  include  the  objectives,  learner  confidence  improves 
and  learner  anxiety  decreases.  Including  objectives  seems  to  be  more 
effective  in  courses  involving  difficult  text  materials  than  in  courses 
with  easier  or  more  understandable  texts. 

W'hile  objectives  may  be  easier  to  write  for  instruction  focused  on 
specific,  concrete  tasks  than  for  more  academic  content  areas  such  as 
history,  no  evidence  suggests  that  objectives  are  more  useful  for  one 
content  area  than  another. 

Expanding  the  task  statements  that  require  instruction  into  objectives 
dictates  specifying  how  the  learner  will  show  learning,  identifying  the 
relevant  conditions  under  which  the  behaviors  are  to  be  exhibited, 
and  specifying  standards  used  to  specify  adequate  performance. 


Reading  List:  Hamilton.  R.  J.  (1985).  A  framework  for  the  evaluation  of  the  effectiveness  of 

adjunct  questions  and  objectives.  Review  of  Educational  Research,  55(1).  47-86. 

Kaplan.  R.,  &  Rothkopf.  E  Z.  (1974).  Instructional  objectives  as  directions  to 
learners:  effect  of  passage  length  and  amount  of  objective  relevant  content. 
Journal  of  Educational  Psychology,  66(3).  448-456. 

Lawton.  J.  T. .  &  Wanska.  S.  (1977).  Advance  organizers  as  a  teaching  strategy:  A 
reply  to  Barnes  and  Clawson.  Review  of  Educational  Research.  7(1).  233-244. 

Melton,  R.  F.  (1978).  Resolution  of  conflicting  claims  covering  the  effects  of 

behavioral  objectives  on  student  learning.  Review  of  Educational  Research.  48(2), 
291-302. 

Rothkopf.  E  Z  .  &  Kaplan.  R.  (1972).  An  exploration  of  the  effect  of  density  and 
specificity  of  instructional  objectives  on  learning  from  text.  Journal  of  Educational 
Psychology.  63,  295-302. 
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Sullivan.  H..  &  Higgins.  N  (1983).  Teaching  for  competence.  New  York:  Teachers 
College.  Columbia  University 


2.  Training  Analysis  Provides  Course  Objectives 


Finding:  Training  needs  analysis  and  task  analysis  yield  objectives  that  serve  as 

the  basis  for  testing  and  designing  instruction. 


Comments:  Course  effectiveness  and  efficiency  depend  on  the  consistency 

between  the  training  requirements,  implied  task  requirements, 
objectives,  task  statements,  and  how  instruction  is  presented. 

The  quality  of  a  training  program  depends  upon  the  adequacy 
of  course  objectives.  Given  an  adequate  analysis  of  training 
requirements,  then  it  is  possible  to  develop  a  relevant  course  and 
adequately  test  its  students. 

Reviews  of  adult  training  courses  reveal  mismatches  between  required 
skills  and  the  objectives  indicating  inconsistency.  Objectives  are 
often  misused  in  the  design  or  presentation  of  the  course  materials. 
Irrelevant  information  may  be  presented.  Required  information, 
as  reflected  in  the  objectives,  is  too  often  missing  from  the  course 
materials. 

The  objectives  determine  what  the  curriculum  and  testing  should 
contain.  Training  requirements  and  objectives  often  specify  one  sort 
of  performance  as  a  goal  of  learning,  but  another  is  tested.  For 
example,  the  objective  “Given  the  weight  of  a  thawed  raw  fowl,  and 
depending  on  whether  or  not  the  fowl  is  to  be  stuffed,  the  student 
will  determine  the  optimum  cooking  time  at  325°F  to  within  three 
minutes"  cannot  be  tested  by  tasting  the  bird,  nor  having  the  student 
recognize  the  answer  on  a  true-false  test.  The  objective  requires  the 
calculation  of  an  answer.  Thus,  a  short  answer  test,  or  fill-in  is  the 
best  test  for  this  objective. 

To  show  the  extent  of  the  problem,  in  an  extensive  study  of  Navy 
technical  training  courses  as  many  as  56%  of  the  training  objectives 
were  found  to  be  inappropriate.  A  major  reason  is  that  the  required 
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training  standards  are  inappropriate;  half  of  the  objectives  were  not 
tested;  less  than  half  of  the  test  items  matched  the  objectives.  These 
courses  can  obviously  be  improved  as,  probably,  can  most  courses. 


Reading  List:  Ellis.  J  A  .  &  Wulfeck  11.  W.  H.  (1980.  March).  Assuring  objective-test  consistency: 

A  systematic  procedure  for  constructing  criterion-referenced  tests  (NPRDC  SR 
80-15).  San  Diego.  CA:  Navy  Personnel  Research  and  Development  Center. 

Ellis.  J  A..  &  Wulfeck  11.  W.  H.  (1986).  Criterion-referenced  measurement 
in  military  technical  training  In  J.  A.  Ellis  (Ed  ).  Military  contributions  to 
instructional  technology  (pp.  60-82).  New  York:  Praeger  Publishers. 

Montague.  W  E..  Ellis.  J.  A  .  &  Wulfeck  11.  W  H.  (1983).  Instructional  quality 
inventory:  A  formative  evaluation  tool  for  instructional  development.  Performance 
and  Instruction  Journal.  22(5).  11-14. 
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FINDINGS  ABOUT  THE  DESIGN  AND  DEVELOPMENT  OF  INSTRUCTION 

Once  objectives  have  been  defined  for  a  course  or  lesson,  a  systematic 
instructional  design  procedure  requires  designing  and  developing  the 
presentation  and  management  of  instruction.  Analysis  and  classification 
ot  the  objectives  is  needed  to  reveal  the  cognitive  operations  involved  in 
performing  the  required  tasks.  Classification  then  aids  in  determining  how¬ 
to  instruct.  Analysis  indicates  essential  and  supportive  training  design  and 
development  requirements.  The  summaries  that  follow'  are  intended  to  assist 
this  process. 
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3.  Focus  on  Post-training  Performance 


Finding:  Conditions  that  yield  rapid  learning  and  good  test  performance  may 

produce  poor  post-instruction  performance  and  transfer. 


Comments:  The  main  objective  of  instructional  programs  is  to  prepare  students 

to  be  able  to  perform  effectively  on  some  task  in  the  real-world. 
However,  having  learned  to  perform  a  task  in  school  does  not 
guarantee  performance  months  later,  in  different  context(s).  Yet. 
those  responsible  for  instruction  usually  see  only  performance  of 
the  learner  during  instruction,  not  on  the  real-world  task(s).  Thus, 
they  may  focus  on  making  learning  efficient  at  the  cost  of  later 
performance. 

Procedures  that  enhance  performance  during  instruction  may  not 
enhance  retention  or  performance  in  different  contexts.  On  the 
other  hand,  procedures  that  introduce  difficulties  for  the  student, 
impairing  performance  during  learning,  can  result  in  durable  and 
adaptable  post-training  performance.  Performance  during  instruction 
may  depend  on  rote  memory,  or  cues  specific  to  the  learning  context 
rather  than  substantial,  durable,  learning 

Thus,  the  effectiveness  of  instruction  should  not  be  measured 
primarily  by  how  fast  students  learn,  or  by  the  level  of  test 
performance  at  the  end  of  instruction.  Instead,  performance  on  the 
post-training  tasks  in  real-world  settings  should  be  used.  To  ensure 
more  adequate  testing,  tests  of  a  student's  progress  during  instruction 
should  measure  performance  as  it  will  be  measured  on  the  real-world 
tasks  in  the  real-world  setting. 


Reading  List:  Druckman.  D,,  &  Bjork.  R.  A.  (1991).  In  the  mind's  eye:  Enhancing  human 
performance.  Washington.  D.  C.:  National  Academy  Press 

Swets,  J..  &  3jork.  R.  (1990).  Enhancing  human  performance:  An  evaluation  of  "New 
Age"  techniques  considered  by  the  U  S.  Army.  Psychological  Science.  1(2).  85-96. 
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4.  Refresher  Training 


Finding:  Systematically  planned  and  monitored  real-world  repeated  testing  and 

rehearsal  of  critical  skills  is  required  for  maintaining  those  skills. 


Comments:  Everyone  loses  trained  skills  and  knowledge  during  extended  periods 

without  specific  exercise  or  practice.  Extended  periods  of  nonuse  are 
common  in  many  real-world  situations. 

Most  adult  training  is  considered  complete  when  students  have 
acquired  minimal  competence.  Partly  this  happens  to  keep  costs 
down.  Presumably,  the  students  are  expected  to  develop  proficiency 
on  "c  job.  However,  many  tasks  are  rarely  exercised.  Therefore, 
procedures  to  maintain  competency  should  be  made  part  of 
performance  requirements.  If  this  can't  be  done,  methods  to 
strengthen  the  original  learning  might  be  implemented  in  training. 
These  methods  may  lengthen  the  training  period,  and  thereby 
increase  costs.  If  refresher  training  is  unfeasible,  as  where  errorless 
performance  is  needed  in  emergencies,  original  learning  might  have 
to  be  carried  to  a  considerably  higher  level. 

Considerable  evidence  shows  that  attention  to  rehearsal  of  rarely 
used  skills  is  often  lacking.  For  example.  Army  researchers  found 
that  few  soldiers  who  performed  basic  soldiering  tasks  adequately 
after  training  could  perform  them  adequately  after  a  year  in  the  field. 
(See  the  example  below.)  Apparently,  they  had  not  performed  or 
rehearsed  most  of  the  skills  during  the  year.  Analysis  of  conditions  in 
the  Navy  also  reveals  occupations  in  which  sailors  do  not  practice  the 
new  skills  for  long  periods.  Such  lapses  promote  skill  and  knowledge 
losses,  not  their  improvement.  Thus,  systematic  management  of  skill 
maintenance  is  needed  to  maintain  competence. 

There  is  no  way  to  make  accurate,  quantitative  predictions  about 
the  rate  of  skiii  loss,  how  fast  relearning  occurs,  or  how  often 
retraining  should  occur.  What  is  known  is  that  initial  learning  during 
training  must  include  the  maximum  amount  of  practice  possible 
under  conditions  that  approximate  the  performance  situation,  and 
that  successive  retraining  or  exercise  sessions  are  needed  at  spaced 
intervals.  Time  management  during  initial  training  must  ensure  that 
time  allotted  is  used  for  practicing  the  skill  and  not  for  irrelevant 
activities  such  as  waiting  for  equipment  or  watching  others  perform. 
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Example: 


Reading  List: 


One  suggestion  is  to  base  the  spacing  of  refresher  practice  sessions 
for  novices  on  how  often  journeymen  perform  the  skill  or  task.  For 
example,  if  journeymen  perform  a  task  monthly  ,  rehearsal  spacing  of 
about  a  month  might  be  advisable  for  novices. 

In  cases  where  rehearsals  are  difficult  or  too  costly  to  arrange,  more 
than  minimal  learning  should  be  provided  during  the  original  course. 
The  main  point  is  that  planning  and  scheduling  the  rehearsal  of 
critical  skills  is  mandatory  . 


Without  systematic  refresher  training,  performance  of  procedural 
skills  declines  rapidly  after  training.  The  following  figure  taken 
from  Hagman  and  Rose  (1983)  and  Shields.  Goldberg,  and  Dressel 
(1979).  depicts  the  decline  in  the  number  of  soldiers  able  to  perform 
basic  soldiering  tasks  adequately  after  training.  The  rate  of  skill  loss 
differs  for  different  tasks,  perhaps  due  to  the  varying  number  of  steps 
in  each  procedure.  In  any  case,  the  decline  suggests  the  need  for 
providing  systematic  practice  to  maintain  skills. 


Bahrick.  Ft.  P  (1979)  Maintenance  of  knowledge:  Questions  about  memory  we 
forgot  to  ask  Journal  of  Experimental  Psychology .  108(3).  296—  308. 

Druckman.  D..  &  Bjork,  R.  (1991)  In  the  mind's  eve:  Enhancing  human 
performance  Washington.  D  C.:  National  Academy  Press. 
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Farr.  M  J.  ( 1987).  The  long  term  retention  of  knowledge  and  s kills  A  cognitive  and 
instructional  perspective.  New  York:  Springer- Verlag 

Flagman.  J  D  .  &  Rose.  A.  M.  (1983).  Retention  of  military  skills:  A  review 
Human  Factors.  25(2).  199-213 

Shields.  J  L  .  Goldberg.  S.  L  .  &  Dressel.  J.  D  (1979.  September).  Retention  of 
basic  soldiering  skills  (Research  Report  1225).  Alexandna.  VA:  US  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences 


5.  Increasing  Time-on-Task 


Finding:  Other  things  being  equal,  providing  supplementary  time  for  practice 

increases  the  level  of  achievement. 


Comments:  One  of  the  simplest  ways  to  increase  student  achievement  is  by 

providing  more  productive  time  for  learning.  Simply  allocating  time 
is  not  sufficient,  the  students  need  to  be  actively  and  productively 
engaged  in  relevant  study.  This  may  be  done  by  providing  a 
variety  of  learning  or  practice  opportunities.  If  training  periods 
are  excessively  long  however,  there  may  be  diminishing  returns  for 
additional  training  time:  the  students  may  become  “burned  out"  and 
accomplish  the  same  amount  of  learning  in  less  time. 

Although  there  is  a  positive  relationship  between  the  amount  of  time 
the  students  spend  on  their  learning  tasks,  the  achievement  gain  is 
modest  compared  to  peer-tutoring  and  other  strategies  for  improving 
achievement. 

There  are  several  consistent  findings  which  have  emerged  from  the 
study  on  instructional  time: 

•  Time  is  a  scarce  resource  —  students  must  have  time  and  the 
opportunity  to  learn. 

•  Time  is  overlapping  on  tasks  and  not  mutually  exclusive;  two 
or  more  activities  can  occur  simultaneously  and  independent 
measurement  of  conceptually  separate  variables  is  difficult  to 
interpret. 

•  Observation  and  interpretation  of  a  student's  time-on-task  and 
the  time  a  teacher  spends  teaching  the  task  may  confuse  different 
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processes.  A  student  may  receive  direct  instruction  relevant  to  the 
objectives  and  also  receive  indirect  instruction  on  them  while  the 
instruction  is  primarily  oriented  toward  loosely-related  objectives. 
Thus,  it  is  difficult  to  assess  time-on-task  or  instruction  time  on 
either  set  of  objectives. 

•  Time-on-task  is  not  the  same  as  time  on  the  right  task. 

•  Allocated  instructional  time  on  a  task  is  not  the  same  as  engaged 
time. 

It  appears  that  all  other  things  being  equal,  the  amount  learned  is 
generally  proportional  to  the  time  spent  in  learning.  Walberg  (1988) 
found  that  30  out  of  35  studies  showed  a  positive  association  between 
learning  performance  and  time  spent  learning. 


Reading  List:  Berliner.  D  C  .  &  Fisher.  C.  W.  (1985).  One  more  time.  In  C.  W.  Fisher  &  D  C 
Berliner  (Eds  ).  Perspectives  on  instructional  time  New  York:  Longman 

Karweit.  N.  L.  (Iy84)  Time-on-task:  A  research  review .  Research  Report  No  333 
Baltimore.  MD:  The  Johns  Hopkins  University.  Center  for  Social  Organization 
of  Schools. 

Leinhardt.  G.  (1985).  Instructional  time:  A  winged  chariot?  In  C.  W.  Fisher  &  D 
C.  Berliner  (Eds.  I.  Perspectives  on  instructional  time.  New  York.  Longman 

Walberg.  H.  J.  (1988)  Synthesis  of  research  on  time  and  learning.  Educational 
Leadership.  45(b).  76-84 


6.  Distributing  Practice  over  Time 


Finding:  Spacing  practice  over  several  sessions  separated  by  periods  of 

unrelated  activities  yields  better  performance  than  equal  amounts  of 
massed  or  concentrated  practice. 


Comments:  For  a  fixed  amount  of  practice,  learning  is  better  if  practice  is 

distributed  in  time.  This  is  generally  true  if  learning  is  measured  by 
performance  on  a  later  retention  test.  If  learning  is  measured  during 
instruction,  massing  practice  yields  better  test  performance.  Since 
instruction  intends  to  optimize  performance  over  the  long-term. 
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this  finding  shows  that  performance  during  instruction  may  be  an 
unreliable  indicator  of  learning.  Sometimes  massed  practice  yields 
long-term  test  performance  less  than  half  that  which  results  from 
spaced  practice.  This  effect,  that  practice  spaced  in  time  results  in 
superior  long-term  performance  than  massed  practice,  is  one  of  the 
most  dependable  findings  from  learning  research. 

Learners  can  absorb  and  integrate  only  a  limited  amount  of  new 
information  at  one  time.  Training  can  be  made  more  effective  by 
designing  shorter  lesson  segments,  and  distributing  them  in  time  by 
separating  them  with  periods  of  other  activities.  Both  non-training 
or  different  training  activities  can  be  interposed  between  scheduled 
sessions.  For  example,  five  classroom  hours  of  lessons  on  quality- 
control  procedures  will  be  learned  best  if  they  occur  on  five 
successive  days,  rather  than  all  on  a  single  day. 

Similarly,  repetition  of  drill  needed  for  developing  skills  can  be 
made  more  effective  by  using  short  sessions  separated  by  other  drill 
activities.  If.  for  example,  trainees  are  learning  code  recognition, 
separating  short  blocks  of  practice  trials  by  practice  on  other  tasks 
or  activities  is  more  conducive  to  learning  than  when  the  practice  is 
massed  together. 

Two  “spaced"  or  distributed  sessions  are  about  twice  as  effective 
as  two  successive  or  massed  sessions,  and  the  difference  between 
them  increases  as  the  number  of  repetitions  increases.  Achievement 
following  massed  practice  sessions  is  often  only  slightly  better  than 
that  following  a  single,  shorter  session.  Massed  presentations  may¬ 
be  unnecessarily  repetitive  and  result  in  learner  boredom.  One 
reason  for  spacings'  efficiency  may  relate  to  the  learners  being 
more  interested  in  material  the  second  time  they  are  exposed  if  the 
repetition  occurs  after  some  delay. 


Reading  List:  Dempster.  F.  N.  (1987).  Time  and  the  production  of  classroom  learning:  Discerning 
implications  from  basic  research.  Educational  Psychologist,  22(1).  1-21 

Druckman.  D..  &  Bjork.  R.  (1991)  In  the  mind's  eye:  Enhancing  human 
performance.  Washington.  D.  C.:  National  Academy  Press. 

Fisher.  C.  W..  &  Berliner.  D.  C.  (Eds.)  (1985).  Perspectives  on  instructional  time. 
New  York:  Longman. 

Rothkopf.  E.  Z..  &  Co'-e.  E.  U.  (1963).  Repetition  interval  and  rehearsal  method 
in  learning  equivalence  from  written  sentences  Journal  of  Verbal  Learning  and 
Verbal  Behavior.  2,  406-416. 
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Smith.  S  M..  &  Rothkopf.  E  Z.  (1984).  Contextual  enrichment  and  distribution  of 
practice  in  the  class-room.  Cognition  and  Instruction,  1.  341-358 

Walbere.  H.  J.  (1988).  Synthesis  of  research  on  time  and  learning  Educational 
Leadership.  45(6).  76-84. 


7.  Development  of  Mental  Models 


Finding:  When  students  learn  to  act  in  accordance  with  a  prescribed  “model"*  of 

performance,  they  develop  conceptual  understanding  that  guides 
competent  performance  more  effectively. 


Comments:  Learning  occurs  by  the  learner's  active  construction  of  schemas  or 

conceptual  structures  that  are  called  “mental  models."  Learners 
construct  and  then  reconstruct  mental  models  that  organize  ideas  and 
their  interrelations.  A  person  makes  use  of  an  internal  model  of  the 
world  to  understand,  explain,  and  predict  things  about  the  world.  If 
people  carry  a  small-scale-model  of  external  reality  in  their  heads, 
they  are  able  to  try  out  various  alternatives,  can  decide  which  is 
the  best  of  them,  react  to  future  situations  before  they  occur,  utilize 
knowledge  of  past  events  in  dealing  with  the  present  and  future. 
Models  allow  people  to  generate  descriptions  of  system  purpose 
and  form,  or  explain  system  functioning  and  observed  states,  and  to 
make  predictions  of  future  states.  These  models  provide  a  means  for 
organizing  and  reorganizing  memory  and  deciding  on  actions. 

Mental  models  evolve  naturally  through  the  interaction  of  the  learner 
and  particular  environments.  They  are  intimately  involved  even  in 
elementary  reading,  mathematics,  and  other  school  subjects.  The 
traditional  view  that  basic  subjects  can  be  taught  as  routine  skills  with 
thinking  and  inferencing  to  follow  later,  provides  a  poor  guide  for 
instructional  practice. 

Unfortunately,  models  developed  by  students  are  often  incorrect  and 
lead  to  errors.  Students  can  be  aided  to  develop  effective  mental 
models  by  providing  them  with  structured  experiences  (such  as 
learning  to  troubleshoot),  by  presenting  functional  representations  of 
systems  (such  as  Kieras'  device  models),  or  by  suggesting  analogies 
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(such  as  tiic  "water  through  a  pipe"  analogy  for  electron  flow). 
Instructional  developers  and  instructors  can  devise  methods  to  help 
students  promote  the  development  of  effective  models.  Instructors 
mav  work  with  students  to  study  how  their  model  systems  work  and 
then  assist  the  student  to  correct  inappropriate  models. 

Students  may  be  helped  to  develop  models  which  represent  the 
functionality  of  the  work  environment,  and  the  devices/equipment 
in  it.  Providing  external  guidance  or  directions,  i.e..  telling  what  to 
do  and  how  to  do  it.  allows  the  build-up  of  experience  coupled  with 
important  cognitive  information  that,  once  internalized,  will  guide 
performance.  An  accurate  mental  model  develops  from  the  way 
events  flow  on-the-job.  how  devices  function  and  can  malfunction, 
and  serves  as  the  scheme  to  guide  personal  action  when  new 
problems  are  encountered.  Having  students  describe  in  detail  the 
steps  they're  using  while  performing,  identifies  errors  and  competence 
develops  faster  and  transfers  readily  to  the  work  environment. 

As  an  example,  take  the  task  of  training  students  to  solve  problems 
in  electric  circuits,  thermodynamics,  or  mechanics.  By  guiding  them 
through  the  steps,  and  explaining  why  they're  taken,  and  then  having 
students  describe  the  factors  and  their  interactions  as  they  solve 
subsequent  problems  they  learn  rapidly  and  accurately.  Instructors 
can  check  the  accuracy  of  a  student's  initial  representation  and 
provide  feedback.  It  focuses  students'  attention  on  the  need  for 
careful  representation  of  all  facets  of  the  problem,  and  provides 
the  basis  for  correct  solutions.  Thus  by  concentrating  on  accurate 
initial  description  of  the  problem,  students  learn  to  internalize  the 
procedures  as  part  of  their  mental  model,  and  use  it  habitually  a- 
approaching  problems  later  on. 


Reading  List:  Gentner.  D  .  &  Stevens.  A.  L.  (1983).  Mental  models  Hillsdale.  NJ:  Erlbaum 
Associates. 

Heller,  J.  I..  &  Retf.  F.  (1984).  Prescribing  effective  human  probicm-soiving 
processes:  Problem  description  in  physics.  Cognition  and  Instruction.  1(2), 
177-216. 


Kieras.  D.  E  (1987).  What  mental  model  should  be  taught:  Choosing  instructional 
content  for  complex  engineered  systems.  In  J.  Psotka.  D  Massey  &  S.  Mutter 
(Eds  ).  Intelligent  tutoring  systems:  Lessons  learned.  Hillsdale.  NJ:  Erlbaum 
Associates. 

Glaser.  R.  (1984)  Education  and  thinking:  The  role  of  knowledge  American 
Psychologist.  39.  91-104 
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8.  Learning  Based  on  Prior  Knowledge 


Finding:  Students  learn  best  when  instruction  is  adapted  to  the  students’ 

existing  knowledge  and  background. 


Comments:  Learning  is  a  process  whereby  students  take  information,  interpret 

it.  connect  it  to  what  they  already  know,  and  if  necessary,  reorganize 
their  mental  models  to  account  for  the  new  findings.  When  dealing 
with  individuals  who  lack  adequate  knowledge  organization  (mental 
models),  we  must  provide  an  initial  or  beginning  knowledge  structure. 
This  may  be  accomplished  by  providing  explicit  organizational 
schemes,  or  temporary  models,  that  serve  as  scaffolds  for  new 
information.  They  form  the  basis  for  organizing  new  knowledge,  and 
provide  a  basis  for  problem  solving. 

Students  begin  learning  new  information  and  concepts  with  a  great 
deal  of  prior  knowledge.  Good  instruction  is  aimed  at  eliciting  this 
prior  knowledge  and  explaining  the  congruence  or  misfit  between 
what  is  to  be  learned  and  the  initial  conceptions.  Learning  progresses 
from  simpler  mental  models  to  more  complex  ones.  Instructional 
materials  need  to  be  designed  to  take  advantage  of  the  student's 
entering  or  existing  knowledge  and  experiences. 

Any  assumed  prior  knowledge  or  skill  for  a  learner  that  is  found  not 
to  be  in  place  at  the  time  of  instruction,  must  be  taught  explicitly  at 
that  time.  The  time,  cost,  and  trouble  of  preparing  students  for  the 
intended  activity  must  be  included  in  the  course  costing. 

Structural  knowledge  enhances  retention  of  the  subject  matter, 
facilitates  problem  solving,  and  leads  directly  to  transfer  to  similar 
and  (perhaps)  new  situations.  Structural  knowledge  may  also  result  in 
intellectual  excitement  and  an  aptitude  for  learning. 


Reading  List:  Bloom.  B.  (1976).  Human  characteristics  and  school  learning.  New  York: 
McGraw-Hill. 

Brophy.  J..  &  Alleman.  J.  (1991).  Activities  as  instructional  tools:  A  framework  for 
analysis  and  evaluation.  Educational  Researcher.  20(4),  9-23. 
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Shavelson.  R  J  (1974).  Methods  for  examining  representations  of  a  subject-matter 
structure  in  a  student's  memory.  Journal  of  Research  in  Science  Teaching.  11(3). 
231-249. 

Walberg.  H  J  .  Strykowski.  B..  Rovai.  E..  &  Hung.  S.  (1984)  Exceptional 
performance  Review  of  Educational  Research.  54(1),  87-112. 


9.  Motivating  Student  Learning 


Finding:  When  instruction  gets  the  learner’s  attention,  is  perceived  as 

relevant,  as  having  attainable  goals,  and  provides  frequent  testing  and 
explanatory  feedback,  students  persist  in  learning,  achieve  well,  and 
enjoy  learning. 


Comments:  There  are  many  factors  that  influence  student  motivation  to  learn 

and  determine  their  achievement.  Techniques  that  make  the  learner 
persist  in  learning,  and  bring  the  learner  into  more  persistent  contact 
with  the  information  or  tasks  to  be  learned,  improve  performance. 
Instructional  design  goals  to  help  accomplish  this  are  to:  make 
learning  interesting,  help  students  set  learning  goals,  promote  student 
tolerance  for  making  errors  while  learning,  and  provide  specific 
feedback  to  guide  students  toward  mastery. 

Instruction  that  is  attractive  and  exciting  is  useful  to  gain  students' 
attention  or  interest.  Therefore,  the  design  of  instruction  would 
attempt  to  stimulate  students’  curiosity  by  including  incongruous,  or 
novel,  or  attractive  material  that  would  make  them  eager  to  engage 
or  study  the  material. 

Students’  understand  the  relevance  of  instruction  when  objectives 
are  explained  to  them  and  new  learning  is  related  to  their  past 
experience  and  knowledge.  Presentations  need  to  explain  the  goals 
of  the  instruction,  how  the  knowledge  is  to  be  used,  and  the  role 
the  learner  will  play  in  the  real-world  when  learning  is  finished.  The 
students  should  feel  the  instruction  meets  their  personal  needs  or 
goals.  Providing  instruction  that  allows  students  to  proceed  through  a 
sequence  of  graded  steps  increasing  the  likelihood  of  success  develops 
confidence  in  their  ability  to  succeed. 


33 


360 


W  E  MONTAGUE  and  F.  G.  KNIRK 


Students  can  learn  to  set  daily  training  goals,  monitor  their  progress 
toward  these  goals,  and  chart  their  progress  to  provide  their  own 
reinforcement.  Instructors  should  ask  their  students  about  their 
progress  toward  these  goals  and  then  provide  positive  verbal 
encouragement  and  reinforcement. 

If  students  fail  to  solve  learning  problems,  they  tend  to  reduce  effort 
expended  in  learning.  Segments  of  instruction  need  to  be  arranged  to 
build  students  expectation  that  they  can  achieve  the  course  goals  with 
sufficient  effort.  Thus,  presenting  simpler  materials  and  problems 
first,  arranging  objectives  in  a  progressive,  logical  sequence,  and 
other  techniques  that  enable  correct  actions  or  explain  what  adequate 
behavior  is  like,  all  motivate  learning.  Student  control  of  the  pacing 
through  the  steps  sometimes  seems  to  increase  learner  confidence. 

Providing  students  with  feedback  explaining  the  adequacy  and 
inadequacy  of  tests  of  learning  and  social  rewards  for  expended 
effort  influences  student  satisfaction.  If  the  instructor  can  establish 
a  trusting  relationship  with  the  student  the  feedback  is  valued 
and  sought.  To  stimulate  student  satisfaction,  praise  for  accurate 
performance,  and  informative  feedback  work  better  than  threats 
or  negative  comments.  Feedback  following  soon  after  performance 
should  emphasize  what  is  or  was  acceptable  about  the  performance. 
Information  correcting  errors  or  guidmg  performance  may  be  most 
useful  given  just  before  another  opportunity  to  perform. 

Instructors  need  to  provide  meaningful  opportunities  for  the  students 
to  use  their  newly-acquired  knowledge  and  skills.  Instructors  should 
make  their  grading  and  performance  requirements  consistent  with 
the  expectations  (criteria  and  standards)  as  stated  in  the  objectives. 
Extrinsic  rewards,  such  as  grades,  scores,  and  points,  while  necessary, 
may  not  motivate  students  as  well  as  goals  and  rewards  based 
on  direct  involvement  with  the  ongoing  training.  Focus  student 
attention  on  long-term  competence  rather  than  extrinsic  rewards. 
Instructional  developers  need  to  develop  training  materials  which 
are  authentic,  life-applicable,  holistic  activities  instead  of  activities 
involving  information  recognition  or  retrieval,  repetitive  seat  work,  or 
isolated  practice  exercises.  Developers  also  need  to  keep  motivational 
materials  relevant  to  the  course  objectives. 


Reading  List:  Brophy.  J  .  &  Allemun.  J  (149])  Activities  as  instructional  toots:  A  framework  for 
analysis  and  evaluation  Educational  Researcher.  20(4),  4-23 

Erickson.  S.  C  (I9N4).  The  essence  of  good  teaching.  San  Francisco:  Josscy-Bass. 
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Horn.  E..  A  Walberg.  H.  (1984).  Achievement  and  interest  as  functions  of  quantity 
jnd  level  of  instruction.  Journal  of  Educational  Research,  77(4).  227-232. 

Johnson.  D  W  et  at.  (1981).  Effects  of  cooperative,  competitive,  and  individualistic 
goal  structures  on  achievement:  A  meta-analvsis.  Psychological  Bulletin.  89(1). 
47-62. 


Keller.  J.  M.  ( 1987)  Strategies  for  stimulating  the  motivation  to  learn.  Performance 
and  Instruction,  26(8).  1-7. 

Lepper.  M..  &  Chabay.  R.  (1985).  Intrinsic  motivation  and  instruction:  Conflicting 
views  on  the  role  of  motivational  processes  in  computer-based  instruction 
Educational  Psychologist.  20,  217-230. 

Malone.  R.  (1981).  Toward  a  theory  of  intrinsically  motivating  instruction.  Cognitive 
Science.  4.  333-364. 


L'rguroglu.  M..  &  Walberg.  H.  (1979).  Motivation  and  achievement:  A  quantitative 
synthesis.  American  Educational  Research.  16(4).  375-389. 


10.  Cooperation  in  Learning 


Finding:  Students  who  help  each  other  and  work  together  learn  more  than 

when  they  study  alone. 


Comments:  Organizing  students  into  small  groups  (of  2-3  students)  to  study 

improves  their  performance  on  achievement  tests.  This  arrangement 
promotes  positive  attitudes  of  students  toward  each  other  and 
toward  learning  and  school.  It  has  potential  to  assist  team  training. 
Group  work  allows  students  to  observe,  imitate  and  learn  from  each 
other,  keeping  each  other  on  task  and  sharing  accomplishments. 

It  is  important  to  make  sure  that  one  student  does  not  dominate 
the  others,  thereby  limiting  their  opportunity  to  learn.  This  can  be 
achieved  by  instituting  procedures  that  make  sure  that  each  student 
spends  an  appropriate  amount  of  time  actively  learning. 

Peer  cooperation  can  take  a  variety  of  forms:  Discussion  groups, 
seminars,  or  tutorial  groups  led  by  teaching  assistants;  the  proctor 
model,  where  senior  students  may  assist  individual  students;  student 
learning  groups  that  are  instructorless  or  self-directed:  or  senior 
students  teaching  entering  students.  Arranging  pe*.r  interaction  in 
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small  groups  to  supplement  regular  classroom  and  laboratory  teaching 
often  helps  slower  and  underachieving  students  to  learn  and  succeed 
in  their  learning. 

Students  coaching  students  raises  their  achievement.  Coaches  benefit 
because  they  learn  more  about  the  material  by  preparing  and  giving 
lessons  to  others.  The  effects  are  greatest  in  long  academic  courses 
and  extensive  drill  and  practice  courses.  Short  courses  that  stress 
test-taking  show  the  least  improvement  from  coaching  methods. 
Classes  that  use  tests  at  the  start  of  the  course  report  stronger 
coaching  effects  than  classes  giving  tests  only  at  the  end. 

Students  bring  many  life  experiences  into  the  classroom,  which  should 
be  acknowledged,  tapped  and  used.  They  can  learn  well  —  and  much 
—  through  cooperative  study  with  respected  peers. 

Students  tend  to  avoid  activities  that  they  believe  will  result  in 
failure.  A  competitive  situation  arouses  the  need  either  to  achieve 
success  or  avoid  failure.  Encouraging  cooperation,  rather  than 
competition,  among  students  promotes  more  effective  learner 
achievement  and  productivity.  Students  competing  for  grades  or 
other  extrinsic  goals  focus  on  beating  other  students  rather  than  on 
understanding  the  course  material  and  learning  how  to  work  as  a 
team  member. 

Studies  of  students  using  computers  for  instruction,  show  that 
students  learn  at  least  as  well  when  paired  at  a  computer  as  they  do 
by  working  by  themselves.  Learner  attitudes  toward  their  class-mates 
also  improves  when  they  are  paired.  Cooperative  learning  promotes 
positive  feelings  of  personal  worth  and  positive  attitudes  toward  the 
course  content. 

Simply  grouping  students  to  work  on  a  learning  task  does  not  ensure 
that  cooperative  learning  occurs  automatically.  Providing  students 
with  a  certain  amount  of  structure  increases  the  liklihood  that  they 
will  work  together  in  a  way  that  contributes  to  their  learning. 


Reading  List:  Bangert-Drowns.  R.  L.,  Kulik,  J.  A.  &  Kuiik.  C.  C.  (1983.  Winter).  Effects  of 
coaching  programs  on  achievement  test  performance.  Review  of  Educational 
Research.  53(4),  571-585. 

Bracey.  G.  W.  (1988).  Two  studies  show  students  gain  when  teaming  up.  Electronic 
Learning.  7(4),  19. 
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Johnson.  D  W..  Johnson.  R  T  .  &  Maruyama.  G  (1983)  Interdependence  and 
interpersonal  attraction  amone  heterogeneous  and  homogeneous  individuals  A 
theoretical  formulation  and  meta-analvsis  of  the  research  Rev ich  of  Educational 
Research.  53,  5—54 

Johnson.  D.  W  .  &  Johnson.  R.  T.  (1986).  Computer-assisted  cooperative  teaming 
Educational  Technology  26(U.  12— IS. 

Slavin.  R  E  (1983).  Cooperative  learning.  New  York:  Longman.  Inc 

Whyte.  M.  M..  Knirk.  F  G  .  Casey.  R.  J..  &  W'lllard  M.  L.  (1991).  Individualistic 
versus  paired  cooperative  computer-assisted  induction:  Matching  instructional 
method  with  cognitive  style  Journal  of  Educational  Technology  Systems.  19(4). 
299-312 


11.  Situated  Teaching  and  Learning 


Finding:  Learning  knowledge  in  a  relevant  context  motivates  learning,  assures 

that  it  is  usable,  and  enables  skillful  performance  of  the  task  in  the 
real  world. 


Comments:  Often,  students  who  have  demonstrated  that  they  know  the  schooled 

facts  and  concepts,  fail  to  use  the  facts  and  concepts  on  appropriate 
occasions.  This  occurs  even  in  technical  schooling.  What  is  taught 
in  schools  resembles  unique  school  activities  rather  than  activities 
useful  for  performing  practical  tasks.  To  ensure  the  usefulness  of 
schooled  knowledge .  instruction  should  be  focused  or  situated  in  a 
context  for  experience  rather  than  be  taught  as  abstract  knowledge 
to  be  applied  later.  Experience  includes  both  the  physical  context  in 
which  a  student's  learning  is  to  be  used,  and  the  tasks,  both  cognitive 
arid  physical,  that  the  student  engages  in  while  in  that  environment. 
Instruction  should  involve  the  same  operations,  the  same  tools,  and 
the  same  machines  (or  their  functional  equivalents)  as  the  practical 
situation. 

Situated  teaching  emphasizes  the  role  of  the  teacher  as  an  expert 
practitioner  who  models  skilled  performance  for  students  and 
inte.  actively  guides  the  development  of  students'  competence 
by  coaching.  What  students  learn  is  focused  in  both  the  external 
and  internal  aspects  of  the  context,  and  can  be  thought  of  as  a 
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Reading  List: 


specific  activity  within  a  specific  culture.  The  closer  instruction  is 
to  actual  technical  performance  in  terms  of  task-related  specificity, 
completeness,  and  realism,  the  greater  its  effectiveness.  Instruction 
should  be  guided  by  level  of  competence  expected  in  the  practical 
situation.  If  a  student  will  be  closely  guided  by  others  (e.g.,  a 
supervisor),  then  the  instructional  program  should  not  expend  the 
time  and  effort  to  graduate  students  who  need  no  guidance.  Effective 
management  of  instruction  matches  the  level  of  instruction  with 
expecteu  conditions  of  performance. 

It  seems  likely  that,  in  most  cases,  schooling  will  be  able  only 
to  approximate  the  authentic  target  context.  Various  forms  of 
representation  may  be  necessary,  depending  on  student  knowledge 
Text  or  other  symbolic  representations  may  be  sufficient  when 
students  have  considerable  background.  When  students  are  less 
knowledgeable,  higher  fidelity  simulation  of  the  practical  situation 
may  be  needed.  There  are  several  important  characteristics  to 
include. 

•  Learning  is  best  situated  in  a  social  context  similar  to  that  in  which 
the  skills  and  knowledge  will  be  used. 

•  The  learner  and  .he  instructor  are  interactive  participants  in  the 
learning  situation:  instructors  model  performance  and  cognition, 
then  observe  and  coach  student  performance. 

•  Normally  covert  cognitive  processes  are  displayed. 

•  Learners  evaluate  their  own  and  others'  performance  (and  vice 
versa)  to  promote  tolerance  for  criticism,  critical  skills,  and 
different  perspectives. 

•  Systematic  experiences  are  arranged  in  a  range  of  conditions. 

•  Materials  must  take  into  account  learner's  frame  of  reference. 


Evidence  from  a  recent  study  (Lajoie  &  Lesgold.  1991)  shows  that  a 
computer-run  version  of  a  situated  practice  environment  works. 
Students  with  20-25  hr  of  practice  in  a  computer  coached  practice 
environment  equalled  the  performance  of  technicians  with  four  years 
of  on-the-job  experience. 


Brown.  J  S  .  Collins.  A..  &  Duguid.  P.  (1989).  Situated  cognition  and  the  culture 
ol  learning  Educational  Researcher.  18,  32-42. 

Collins.  A  .  Brown.  J.  S.,  &  Newma.i,  S.  E.  (1989)  Cognitive  apprenticeship: 

Teaching  the  craft  of  reading,  writing,  and  mathematics  In  L.  B.  Rcsnick  (Ed). 
Know  mg.  learn  mi;  and  instruction:  Essays  in  honor  of  Robert  Glaser  Hillsdale. 
NJ:  Erlbaum. 
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Hiinchotn.  P  (  Dulls  I  M  .  A  f-ishman.  B  J  (19921  Constructivism  and  the 
dcsittn  o!  learning  environments:  Context  and  authentic  activities  lor  lear..inj> 

In  I  M  Duffy .  J  Lowyck  A  D  Jonassen  (Bus  ).  Designing  cm irtmments  for 
conuructn  e  Earning  Hillsdale.  NJ  Erlbaum  A  Associates 

l-ajoie.  S  P  .  A  Lesgold.  A  (1992)  Apprenticeship  training  in  the  workplace 
Computer-coached  practice  environment  us  a  new  form  of  apprenticeship  In 
M  J  Parr  A  J  Psotku  (Eds  ).  Intelligent  instruction  h\  computer  (Chapter  1,  pp 
15-36)  Washington,  D  C  Taylor  A  Francis 

M  orris.  N  M  .  A  Rouse.  W  (|W5)  Review  and  evaluation  of  empirical  research 
on  troubleshooting  Human  Factors.  27,  503-530 

Sticht.  T.  G  .  Armstrong.  W  B  .  Hickey.  D  T  .  &  Caylor.  J  S  (I9K7)  Cast  otf 
youth  Policy  ami  training  methods  from  the  military  experience  New  York 
Praeger 


12.  Student  Control  of  Learning 


Finding:  Students’  perception  of  who  controls  the  key  events  in  learning 

significantly  affects  their  academic  achievement. 


Comments:  In  the  classroom,  students  generally  attribute  their  learning  success 

to  a  combination  of  ability,  effort,  task  difficulty,  and  luck.  They 
believe  that  if  they  can  significantly  control  their  learning,  they  can 
also  organize  their  environment  for  maximum  success;  that  is  they  can 
"make  their  own  luck." 

Research  has  repeatedly  demonstrated  that  learning  disabled  and 
other  slow  students  tend  to  think  that  other  individuals  cause 
their  successes  and  failures.  Successful  students  are  more  likelv  to 
recognize  their  responsibility  for  their  achievement.  According 
to  recent  evidence,  student  perceptions  about  who  caused  their 
successes  and  failures  depend  on  situational  factors.  Certainly 
instructors  can  change  these  perceptions.  Feedback  pointing  out 
the  quality  of  performance  and  how  to  improve  it  can  teach  slower 
students  to  recognize  that  they  arc  responsible  for  their  learning  and 
performance. 
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Reading  List:  Bannister.  B  D  (1986)  Performance  outcome  feedback  and  attributional  feedback 
Interactive  effects  on  recipient  responses.  Journal  of  Applied  Psychology ,  71(2). 
203-210. 

Bar-Ta),  D.  (1978)  Attributional  analysis  of  achievement-related  behavior.  Review  of 
Educational  Research.  48(2).  2S9-271. 

Licht.  B.  G..  Kistner.  J.  A..  Ozkeragoz,  T.,  Shapiro.  S.,  &  Clausen.  L.  (1985) 

Causal  attributions  of  learning  disabled  children:  Individual  differences  and  their 
implications  for  persistence.  Journal  of  Educational  Psychology.  77(2),  206-216. 

Schunk.  D  H.  (1985)  Self-efficacy  and  classroom  learning  Psychology  in  the 
Schools.  22.  208-223. 

Stipek.  D  J  .  &  Weisz.  J.  R.  (1981).  Perceived  personal  control  and  academic 
achievement.  Review  of  Educational  Research.  51(1),  101-137. 


13.  Writing  Text  Materials 


Finding:  Enhancement  of  text  in  books  or  manuals  through  orientation, 

summaries,  examples  and  diagrams  can  aid  student  comprehension 
and  learning. 


Comments:  Much  training  is  accomplished  through  written  descriptions  or 

discussion.  Texts  are  prepared  to  serve  as  a  basis  for  student 
learning  by  providing  facts,  examples,  and  explanations.  To  learn, 
students  must  understand  the  materials  and  how  they  can  apply 
the  information.  Descriptions,  instructions,  and  explanations  are 
often  difficult  to  understand  because  of  terminology,  inadequate 
connections  to  student  knowledge,  or  a  “topic-orientation'''  that  tells 
all  about  a  subject,  but  not  “what  a  person  does”  or  “how  to  do  it.” 

Designers/writers  need  to  be  aware  of  student  vocabulaty  and 
knowledge  limitations.  The  designers  should  also  use  techniques 
that  will  help  the  students  to  comprehend  and  remember  the  course 
materials. 

Writing  should  be  performance-oriented,  rather  than  topic-oriented. 
Topic-oriented  writing  looks  like  reference  material  aimed  at  a 
general,  unspecified  audience,  telling  all  about  a  subject  and  not 
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how  to  apply  the  information.  Performance-oriented  writing  focuses 
on  specific  users;  describes  their  roles,  tasks,  and  responsibilities; 
and  gives  them  information  they  need  about  how  to  perform.  The 
advantage  of  performance-oriented  text  is  that  readers  do  not  have  to 
infer  and  conceptualize  what  to  do;  it  is  stated  explicitly. 

The  materials  need  to  be  explicit.  Explicit  instruction  will  result 
in  the  students  demonstrating  significantly  greater  metacogmtive 
awareness  of  the  content.  Cues  should  be  provided  about  what  is 
going  to  be  learned,  guided  practice  of  the  to-be-learned  material 
should  be  included  and  application  examples  are  desirable. 

Several  techniques  can  be  used  to  improve  student  comprehension 
of  text: 

•  Pre-presentation  summaries  or  “advance  organizers"  outline  what 
is  to  be  learned,  provide  structure,  and  improve  learning. 

•  Inserting  pictures  showing  spatial  relationships,  object  form,  or 
internal  structure  can  be  powerful  aids  to  comprehension. 

•  Concrete  examples  can  be  used  to  clarify  abstract  ideas  or  depict 
how  principles  work. 

•  Methods  that  put  demands  on  the  trainee  in  reading  and 
"processing"  the  text  are  especially  useful. 

•  Questions  inserted  before  or  after  text  segments  can  help  students 
to  identify  important  information,  and  make  desired  inferences. 

•  Asking  students  to  relate  new  information  to  what  they  already 
know  or  to  paraphrase  the  content  can  aid  learning. 

•  Writers  can  ask  students  to  construct  a  diagram  or  "map" 
depicting  the  relationship  of  ideas  in  text  to  aid  comprehension  and 
remembering  the  information. 

The  insertion  of  interesting  details  in  texts  appears  to  divert  attention 
from  important  generalizations  in  the  text  to  the  interesting  and  often 
irrelevant  detail.  Placement  of  the  details,  however,  does  not  affect 
learner  recall  and  appears  to  improve  how  interesting  the  text  is  — 
especially  if  the  learners  knew  little  about  the  topic  being  learned.  It 
is  probable  that  cartoons  or  other  humorous  insertions  have  similar 
effects.  If  the  students  lack  interest  in  a  course,  or  topic,  the  use 
of  interesting  materials  should  increase  student  attentiveness  and 
attitudes  toward  the  course. 


Reading  List:  Gagne.  R.  M.  (1986).  Instructional  technology:  The  research  field.  Journal  of 
Instructional  Development.  8(3).  7-14. 

Garner,  R..  Alexander.  P  A  .  Gillingham.  M.  G..  Kulikowich.  J.  M..  &  Brown.  R. 
(1991).  Interest  and  learning  from  text.  American  Educational  Research  Journal. 
28(3).  643-659. 
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Jonassen.  D  H.  (1985)  The  technology  of  texi.  Volume  2:  Principles  for  structuring, 
designing  and  displaying  text.  Englewood  Cliffs.  NJ:  Educational  Technology 
Publications 

Kern.  R  P  .  Sticht .  T.  G..  Weltv.  D..  &.  Hauke.  R  N.  (1976.  November). 

Guidebook  for  the  development  of  Army  training  literature  Alexandria,  VA;  U  S 
Arms  Research  Institute  for  the  Behavioral  and  Social  Sciences. 


Nueent.  G.  C.  (1982)  Pictures,  audio,  and  print:  Symbolic  representation  and  effect 
on  learning  Educational  Communication  and  Technology  Journal.  30,  163-174. 


14.  Readability  of  Instructional  Materials 


Finding:  Readability  scores  indicate  approximately  how  much  difficulty  students 

will  have  in  reading  or  listening  to  training  materials. 


Comments:  Readability  formulas  predict  how  well  people  of  varying  reading 

ability  can  recall  text  they  have  read  or  heard.  However,  their 
usefulness  for  predicting  comprehension  of  instruction  is  limited 
because  they  do  not: 

•  Provide  precise  estimates  of  difficulty. 

•  Provide  estimates  of  difficulty  for  non-text  materials  like  tables  and 
figures  that  make  up  much  of  the  instruction  in  technical  training 
courses. 

•  Take  into  account  how  the  text  materials  will  be  used,  for 
example,  whether  they  are  studied  and  learned  or  read  while 
performing. 

•  Take  into  account  students’  background  knowledge  in  the  area 
and  related  areas.  Students  with  a  lot  of  background  knowledge 
can  attain  high  comprehension  while  having  reading  ability  several 
grade  levels  lower. 

•  Provide  specifications  for  writing  readable  materials;  one  should 
not  write  to  the  formula. 

Issues  other  than  readability  should  be  considered  in  developing 
instruction.  For  example,  performance-oriented  text  is  recommended 
in  manuals  over  topic-orientation.  Topic-oriented  text  tells  the  reader 
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everything  you  want  to  know  about  the  topic,  but  it  does  not  tell 
what  action(s)  are  to  be  performed.  A  reader  must  infer  what  to  do. 
Performance-oriented  text  explicitly  tells  the  reader  what  actions  are 
expected  of  them.  Surprisingly,  technical  manuals  and  texts  are  often 
topic-oriented. 


Reading  List:  Dole.  J  A  .  Duffy.  G.  G.,  Roehler.  L.  R  .  &  Pearson.  P.  D.  (1991)  Moving  from 
the  old  to  the  nets:  Research  on  reading  comprehension  instruction.  Review  of 
Educational  Research.  61(3).  239-264. 

Duffs  .  T..  &  Waller.  R.  (Eds.)  (1985)  Designing  usable  lexis.  Ness  York:  Academic 
Press. 

Hurtles.  J  (19~S)  Designing  instructional  text.  Ness  York:  Nichols 

Kern.  R.  (1979).  Usefulness  of  readability  formulas  for  achieving  Army  readabihti 
objectives:  Research  and  state-of-the-art  applied  to  the  Army's  problem  Fort 
Benjamin  Harrison.  ID:  Technical  Advisory  Service.  U.S.  Army  Research 
Institute 

Kern.  R  .  Sticht.  T.,  Welty.  D..  &  Hauke.  R.  (1977).  Guidebook  for  the 

development  of  Army  training  literature  I  P-75-3).  Alexandria.  VA:  US.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences 

Sticht.  T  (1972)  Learning  by  listening.  In  R.  Freedle  &  J.  Carroll  (Eds  ).  Language 
comprehension  and  the  acquisition  of  knowledge.  Washington.  D  C  :  V.  H 
Winston  &  Sons. 

Sticht.  T  (1985)  Understanding  readers  and  their  uses  of  text.  In  T.  M  Duffy  & 

R.  Waller  (Eds  ).  Designing  usable  texts.  New  York:  Academic  Press. 


15.  Memorization  Aids 


Finding:  Mnemonic  devices  or  coding  systems  help  students  recall  important 

information  when  needed. 


Comments:  Learning  by  rote  seems  an  inefficient  way  of  remembering.  When 

people  arc  faced  with  a  rote  memory  task,  they  often  try  to  devise 
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Example: 


some  scheme  to  make  the  learning  task  easier.  Therefore,  teaching 
students  various  types  of  memory  devices  or  mnemonic  procedures 
has  often  been  proposed  to  make  remembering  materials  easier  A 
pictorial  mnemonic  is  also  more  effective  than  a  figural,  or  schematic, 
taxonomy  for  lower  recall  and  higher-order  thinking  skills. 

Aids  to  memorization  can  take  various  forms  which  can  be  given  to 
students.  Some  formal  devices  used  in  training  which  provide  students 
with  mental  cuing  structures  are  made  up  of  visual  images  or  words 
in  sentences  or  rhymes  that  mediate  between  a  signal  to  the  learner 
to  recall  and  the  information  to  be  recalled.  Students  learn  the  cuing 
structure  first  and  associate  each  item  of  new  information  with  one  or 
more  of  the  already  memorized  cuing  structure.  Later,  they  use  the 
structure  for  recall  through  a  self-cuing  process. 

Usually,  the  cuing  structure  is  not  conceptually  related  to  the 
information  it  cues.  For  example,  consider  the  rhyming  peg- word 
mnemonic  system  “One  is  a  bun;  Two  is  a  shoe;  Three  is  a  tree; 
etc...”.  Students  first  memorize  the  ordered  rhymes.  Then,  when 
they  must  learn  an  arbitrary  set  of  items  in  order,  they  relate  the  first 
with  "bun,"  the  second  with  “shoe”,  and  so  on.  Instructions  often 
suggest  using  visualization  to  help  relate  the  items.  Mnemonic  devices 
are  effective  in  helping  students  to  recall  unorganized  names  and 
procedural  data. 

Students  can  be  encouraged  to  devise  their  own  mnemonic  devices 
such  as  a  story  to  help  recall  a  list  of  arbitrary  words  or  phrases. 

Ship  handlers  have  to  remember  colors  of  signal  lights  associated 
with  marking  intervals  in  distances  between  ships:  for  example,  red 
(20  yards),  yellow  (40  yards),  blue  (60  yards),  white  (80  yards),  green 
(100  yards).  To  remember  the  r  y  b  w  g  sequence,  students  are  asked 
to  make  up  a  sentence  (or  are  given  one).  Thus,  “rub  your  belly  with 
grease"  encodes  the  sequence,  is  memorable,  and  facilitates  recall  of 
the  information.  Mnemonic  facilitation  of  memory  results  from  the 
use  of  visuals  which  have  meaning  to  a  learner,  can  be  retrieved  as 
visual  images,  and  thus  facilitate  recall. 


Normally,  individuals  can,  in  the  short-term,  recall  a  string  of  about 
seven  unrelated  items  like  digits  or  letters  presented  to  them  one  at 
a  time.  However,  recall  performance  can  be  improved  to  many  times 
that  level  by  using  a  learning  strategy  like  coding  items  into  more 
meaningful  chunks,  and  by  practicing  a  lot. 
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The  figure  shows  data  obtained  from  two  persons  who  learned  a 
way  to  increase  memory-span  to  exceptional  performance  levels 
(Ericsson  &  Chase.  1982).  The  lighter  line  is  that  for  a  person  who 
was  read  strings  of  digits  and  simply  asked  to  recall  them.  He  was 
a  runner,  knew  much  about  running,  running  times  and  records 
for  competition.  Some  digit  groups  reminded  him  of  running  times 
for  different  races,  so  he  began  to  code  3-  and  4-digit  groups  as 
running  times  for  various  races  (e.g.,  3  4  9  2  was  coded  as  3  minutes 
49.2  seconds,  near  the  world  record  time  for  a  mile).  As  practice 
continued  he  constructed  other  mnemonic  associations  along  with  the 
times,  such  as  ages  and  dates.  With  the  development  of  this  coding 
scheme  he  was  able  to  recall  about  80  digits  accurately  after  abou* 

220  practice  sessions.  Another  runner  was  taught  the  memorization 
scheme.  His  performance  on  over  275  practice  sessions  is  shown  as 
the  darker  line.  Both  performed  exceptionally.  The  important  point 
is  that  coding  schemes  based  on  a  person's  existing  knowledge  can 
serve  as  learning  strategies.  Self-generated  schemes  are  powerful  tools 
in  learning.  Good  ones  can  be  useful  to  other  learners,  and  provide 
them  an  early  boost  in  performance.  Furthermore,  extensive  practice 
is  necessary  to  develop  skill. 


Reading  List:  Bellezza.  F.  S.  (1981).  Mnemonic  devices:  Classification,  characteristics  and  criteria. 
Review  of  Educational  Research,  51(2).  247-275. 

Braby.  R..  Kincaid.  J.  P  .  &  Aagard.  J.  A.  (1978.  July).  Use  of  mnemonics  in 

training  materials:  A  guide  for  technical  writers  (TAEG  Report  No.  60).  Orlando. 
FL:  Training  Analysis  and  Evaluation  Group. 
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Ericsson.  K.  A..  &  Chase.  W  G  (1982).  Exceptional  memory  American  Scientist. 
70(0) 

Levin.  M  E  .  &  Levin.  J  R.  (1990)  Scientific  mnemonomies:  Methods  for 

maximizing  more  than  memory  American  Educational  Research  Journal.  27(2). 
301-321 

Pressley.  M  .  Levin.  J..  &  Delaney.  H.  (1982).  The  mnemonic  keyword  method 
Renew  of  Educational  Research.  52,  61-91. 

Williams.  S.  S.  (1979).  Improving  memory  :  Developing  imagery  skills  in  returning 
adult  students.  Adult  Education.  51(5).  286-92. 


16.  Using  Examples  and  Nonexamples 


Finding:  Providing  students  with  representative  good  examples  and  contrasting 

them  with  bad  examples  teaches  them  desired  knowledge  and  skills. 


Comments:  A  problem  when  developing  instruction  is  presenting  of  the  subject 

matter  to  be  learned  in  a  form  that  promotes  student  learning.  One 
technique  for  accomplishing  this  goal  involves  presenting  contrasting 
examples  that  are  accurate  or  positive  representations  and  those  that 
might  be,  but  are  nonexamples.  Nonexamples  refine  the  definitions  or 
concepts  being  investigated  by  the  learner.  The  contrast  develops  the 
learners'  representation  or  knowledge. 

It  is  necessary  to  collect  a  variety  of  examples  that  are  not  ambiguous 
or  confusing  which  illustrate  the  task  so  the  student  will  understand 
the  problem  being  studied  and  not  acquire  misconceptions.  Each 
example  must  be  complete  and  self-contained.  Each  should  contain 
the  necessary  critical  features,  or  attributes,  so  the  student  can 
observe  their  presence  or  absence  to  construct  adequate 
generalizations  or  representations  of  the  task. 

The  form  and  fidelity  of  each  example  is  important  because  it  must 
adequately  represent  the  critical  features  of  the  task.  Examples 
should  be  as  divergent  as  possible  while  belonging  to  the  task  being 
taught.  This  is  done  to  prevent  irrelevant  features  from  being 
encoded  into  the  generalization  while  facilitating  the  formation  of 
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appropriate  schema.  Avoid  extreme  variations  as  they  make  examples 
difficult  to  understand  or  demand  skills  the  student  may  not  have. 
Easier  examples  should  be  provided  early  in  the  lesson  with  a  gradual 
increase  in  difficulty. 

It  is  important  to  use  attention  focusing  devices  to  direct  student 
attention  to  critical  features,  to  the  confusing  features  and  to  the 
absence  of  critical  features.  Learners  tend  to  respond  to  similar  sets 
of  stimuli  in  similar  ways  even  when  the  response  may  be  incorrect 
in  one  situation.  Discrimination  is  facilitated  by  exposing  learners  to 
examples  paired  with  appropriate  nonexamples  which  focus  on  the 
critical  differences  so  they  may  be  easily  identified. 

Just  as  students  learn  from  their  mistakes,  they  are  not  harmed 
by  examples  which  do  not  reflect  the  instruction:  instead  these 
nonexamples  refine  and  clarify  definitions  and  illustrations. 


Reading  List:  Bloom.  B.  S.  (1968).  Learning  for  mastery.  In  Evaluation  comment.  1(2).  Los 
Angeles:  University  of  California  at  Los  Angeles.  Center  for  the  Study  of 
Evaluation  of  Instructional  Programs. 

Gage.  E.  D.  (1985),  The  cognitive  psychology  of  school  learning.  Boston:  Little. 
Broun  and  Co. 

Gagne.  R  M.  (1985)  The  conditions  of  learning  and  theory  of  instruction.  New 
York:  Holt.  Rinehart  and  Winston 

Merrill.  M  D.  (1987).  A  lesson  illustrating  component  display  theory  In  C.  M. 
Reigeluth  (Ed  ).  Instructional  design  theories  m  action.  Hillsdale.  NJ:  Lawrence 
Erlbaum. 

Merrill.  M  D..  &  Tennyson.  R.  D  (1977)  Teaching  concepts:  An  instructional 
design  guide.  Englewood  Cliffs.  NJ:  Educational  Technology  Publications 

Reigeluth.  C.  M.  (Ed.)  (198?)  Instructional  design  theories  and  models:  An  overvie h 
of  their  current  status  Hillsdale.  NJ:  Lawrence  Erlbaum. 
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FINDINGS  ABOUT  THE  PRESENTATION  OF  INSTRUCTION 

Following  course  requirements  and  content  analysis,  the  design  of  instruction 
and  the 

determination  of  presentation  mode  should  occur.  Presentation  decisions 
involve  instructor  availability  and  skills,  student  study  and  listening  skills, 
homework  requirements  and  testing  design. 
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17.  Instructor  Classroom  Leadership 


Finding:  Effective  instructor  leadership  in  the  classroom  promotes  effective 

student  learning. 


Comments:  Instructors  lead  students  to  learning  by  focusing  on  performance. 

presenting  well  conceived  learning  objectives,  conducting  regular, 
and  comprehensive  evaluations  of  student  learning,  having  high 
expectations  of  all  students,  and  providing  a  purposeful  and  peaceful 
learning  environment. 

Instructors  should  protect  minority  opinions,  keep  disagreements 
under  control,  point  out  relationships  between  various  opinions  and 
ideas,  and  remind  the  class  of  the  variety  of  potential  solutions  to  a 
problem. 

Instructors  can  observe  each  other  in  the  classroom  and  comment  on 
their  observations.  This  constructive  feedback  can  help  the  observed 
instructor  become  more  effective  and  improve  morale. 

Good  classroom  management  is  essential  for  classes  with  problem 
students  such  as  those  who  are  consistent  underachievers,  hostile, 
aggressive,  defiant,  easily  distracted,  socially  withdrawn,  or  rejected 
by  the  other  students. 

Instructors  should  give  grades  that  reflect  the  student's  skill  or  the 
degree  to  which  they  have  achieved  the  course  objectives  as  measured 
by  a  criterion  test.  Grades  should  not  be  used  as  disciplinary  tools. 
When  students  actively  participate  in  their  learning,  disciplinary 
problems  are  reduced. 

Instructors  must  help  students  perceive  the  instruction  as  relevant  and 
interesting,  reinforce  good  behavior,  seek  friendly  personal 
relationships  with  the  students,  help  them  develop  a  sense  of  right 
and  wrong,  and  encourage  them  to  cooperate  with  other  students 
and  staff. 


Reading  List:  Brophy.  J.  E.  (1985).  Classroom  management  as  instruction:  Socializing  self-guidance 
in  students.  Theory  into  Practice,  24(4).  233-240. 
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Cohen.  M  f  1^82).  Effective  schools:  Accumulating  research  findings  American 
Education.  18(1 1.  13—  Its 

Gold  M  .  A  Mann.  D  (1W4).  Expelled  to  a  friendlier  place  A  study  of  effect! i c 
aln-rnatue  schools  Ann  Arbor,  Ml:  The  Umversitv  of  Michigan  Press. 

Zemkc.  R  S.  (19S1.  June)  30  Things  we  know  for  sure  about  adult  learning 
Trammel HRD.  18(b).  45-52. 


18.  Instructor  Classroom  Role 


Finding:  Student  activities  during  learning  are  more  important  in  determining 

what  is  learned  than  the  instructor's  presentation.  Instructors  aid 
student  achievement  by  getting  students  to  engage  in  activities  that  are 
likely  to  result  in  learning. 


Comments:  Typical  classroom  instruction  often  places  students  in  a  passive  role 

(such  as  listening  or  watching),  where  they  learn  less  than  when  they 
are  actively  involved. 

Effective  instructors  do  not  merely  state  facts  and  ideas:  they  know 
how  to  get  students  actively  engaged  in  appropriate  learning  activities 
for  attaining  the  desired  outcomes.  Learning  is  an  active  process 
in  the  learner:  the  instructor's  task  therefore  involves  more  than 
dissemination  of  information.  Instruction  must  consider  factors  such 
as  prior  knowledge,  the  context  in  which  the  material  is  presented, 
the  uses  intended  for  the  outcomes,  and  the  realization  that  student 
understanding  of  new  information  depends  on  how  well  it  relates  to 
their  prior  knowledge. 

Students  often  begin  learning  with  substantial  misconceptions  about 
the  material  they  are  studying  and  its  intended  use.  Even  students 
who  get  high  grades  have  these  misconceptions.  Students  also  make 
systematic  errors  owing  to  misconceptions  or  erroneous  procedures 
based  on  their  current  knowledge.  The  instructor  needs  to  address 
the  inadequate  prior  knowledge  directly  and  present  instruction  likely 
to  remove  the  misconceptions  and  faulty  information.  The  instructor 
must  understand  how  current  and  prior  knowledge  determines 
what  the  students  will  learn  from  new  material  that  conflicts  with 
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their  existing  beliefs.  Students  should  be  asked  to  reveal  their 
misconceptions  so  that  the  instruction  can  confront  them. 

Instructors  should  set  and  communicate  high  expectations  about  what 
students  can  or  cannot  learn.  Ins’ructor  expectations  often  become 
self-fulfilling  prophecies  as  students  tend  to  meet,  but  not  exceed, 
instructor  expectations.  Too  frequently  instructors  communicate 
low  expectations  of  students  by:  (1)  seating  them  in  the  back  of  the 
classroom.  (2)  seldom  calling  on  them  to  answer  oral  questions,  and 
(3)  allowing  them  to  do  less  work. 

Instructors  often  function  better  as  facilitators  of  student  learning 
than  in  their  traditional  role  as  presenters  of  readv-madt  informatics. 
In  a  cognitively  based  view  of  comprehension-oriented  instruction, 
the  teacher  is  a  mediator  who  assists  learners  to  construct 
understanding  about  (1)  the  content  of  the  lecture-text,  (2)  strategies 
for  interpreting  thei*-  instruction,  and  (3)  the  nature  of  the  learning 
process  (reading,  listening)  itself. 


Reading  List:  Brophv.  J.  E.  ((981).  Teacher  praise:  A  functional  analysis.  Review  of  Educational 
Research,  51.  5-32. 

Dole.  J.  A  .  Duffv.  G.  G..  Roehler.  L.  R..  &  Pearson.  P  D.  (1991).  Moving  from 
the  old  to  the  new:  Research  on  reading  comprehension  instruction.  Review-  of 
Educational  Research.  61(2).  239-264. 

Office  of  Technology  Assessment  (1988).  Power  on!  New  tools  for  teaching  and 
learning.  Washington.  D.C.:  U.S.  Government  Printing  Office. 

Shuell.  T  J.  (1986).  Cognitive  conceptions  of  learning.  Review  of  Educational 
Research.  56(4).  411-436 

Snow.  R.  E  .  Lowman.  D.  F.  ( 1984).  Toward  a  theory  of  cognitive  aptitude  for 
learning  from  instruction  Journal  of  Educational  Psychology .  76,  347-376. 


19.  Teaching  Students  How  to  Learn 


Finding:  The  ways  students  study  influence  what  and  how'  much  they  learn. 

Students  can  learn  effective  study  strategies. 


51 


W  E  MONTAGUE  ami  F.  G.  K.N1RK 


Comments: 


Example: 


Good  students  have  been  found  to  use  study  strategies  which  can 
be  taught  to  others.  Study  or  learning  strategies  may  affect  learner 
motivation  or  the  way  they  select,  acquire,  organize,  or  integrate  new 
knowledge.  For  example,  learners  may  coach  themselves  to  reduce 
anxiety,  use  imaging  to  relate  vocabulary  words  and  meanings,  or 
summarize  and  take  notes  to  memorize  written  material. 

Average  and  low  ability  students  use  these  strategies  less  than  high 
ability  students.  Average  ability  students  can  learn  the  skills;  however 
low  ability  students  may  need  to  be  taught  when,  as  well  as  how. 
to  use  these  strategies.  Once  they  have  learned  the  strategies,  ail 
students  can  study  and  learn  more  efficiently,  but  they  often  need  to 
be  encouraged  to  do  so. 

Monitor  and  adjust  the  way  they  study  based  on: 

•  Whether  they  understand  difficult  material. 

•  How  much  time  they  have  for  studying. 

•  How  much  they  know-  about  the  material. 

•  The  standards  they  must  meet. 

•  Space  study  sessions  on  a  topic  over  available  time  and  not  to 
work  continuously  on  a  single  topic. 

Use  the  study  strategies  appropriate  for  the  learning  task.  For 
example,  use  rehearsal  and  self-testing  to  memorize  ordered  lists, 
take  notes  that  paraphrase  a  lecture,  or  organize  information  in  text 
by  identifying  the  main  ideas  and  relating  them  to  prior  or  current 
knowledge. 

Assess  their  progress  bv  frequent  se’f-questioning  and  modify  the 
strategies  as  needed. 


Training  Students  How  to  Learn  from  Text:  Training  in  techniques  for 
learning  from  text  materials  har  a  substantial  effect  on  performance 
on  tests  covering  the  content  studied. 

Students  given  training  in  how  to  study  text  material  outperform 
students  not  given  training.  For  example,  they  were  taught  how 
to  make  a  network  map  of  the  information  in  the  text,  a  spatial 
representation  cf  the  information  and  how  to  paraphrase,  to  draw 
pictorial  representations  of  ideas  and  concepts  in  the  network.  Four 
different  measures  were  used  to  examine  the  effect  of  the  training. 

As  can  be  seen  in  the  figure,  the  trained  students  substantially 
outperformed  untrained  ones  on  essay  and  short-answer  tests.  The 
histogram  bars  show  how  much  the  scores  of  the  trained  students 


52 


Wh.H  Work'  id  Adult  instruction 


exceeded  those  ot  untrained  students  On  a  “cloze "  test  even  nth 
word  in  the  material  is  deleted,  and  the  student  tries  to  till  mi  the 
correct  word  from  mentors  Trained  students  showed  superior 
performance  on  that  type  of  test  also  On  a  multiple-choice  test 
trained  students'  superiority  was  slight.  This  type  of  test  is  not  as 
useful  a  test  for  examining  student  learning  and  understanding 


Reading  List:  Dansereau.  I)  F  (1985)  Learning  strategy  research  In  J  Segal.  S  Chipman  &  R 

(ilascr  (Eds  ).  Thinking  and  /earning  skills.  Vol  I  Relating  instruction  to  research 
(pp  2(19-240)  Hillsdale.  NJ  Erlbaum  Associates 

Dansereau.  D  F  .  Brooks.  I  W  .  Holley.  ('  D  .  &  Collins.  K  W  <I9K.)| 
l  earning  strategics  training  F.ffeets  of  sequencing  Journal  of  Experimental 
Education .  51(.'l  102- ION 

Dobrosolny.  J  L  .  McComhs.  B  L  .  &  Judd.  W  A  (1980.  March!  Studs  skills 
package  Hex  elopmrnt  and  evaluation  ( AF'HRL-TR-79-43)  Brooks  AFB.  TX  Air 
Force  Human  Resources  Laboratory 

Weinstein.  C  E..  &  Mayer.  R  E  (1986)  The  teaching  of  learning  strategies  In 
M  C  Wittrock  (Ed  ).  Handbook  of  research  on  teaching  (3rd  cd.)  New  York 
Macmillan  Publishing  Company 
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20.  Instructor  Presentation  Stimulates  Learning 


Finding:  Students  perform  best  when  their  instructors  inspire  them  to  take  an 

active  role  in  their  learning. 


Comments:  Good  instructors  are  subject  matter  experts  in  what  they  teach,  are 

well  prepared  for  student  questions,  and  stimulate  student  interaction. 
Because  students  can  remember  only  a  small  amount  of  material 
presented  orally  or  visually,  emphasizing  the  systematic,  logical 
structure  of  the  material  can  help  students  learn  and  remember.  It  is 
also  advisable  to  present  no  more  than  two  or  three  main  ideas  in  a 
15  min  segment. 

Instructors  can  use  techniques  to  stimulate  students  to  ume  an 
active  role  in  understanding  what  is  taught.  For  example,  students 
learn  best  when  they  receive  summaries  of  the  main  ideas  or  goals 
of  the  presentation.  Reasons  for  learning  the  information,  and 
illustrations,  tables,  and  charts  can  be  supplied  for  later  study 

Instructors  who  ask  questions  and  present  problems  force  students 
to  think  of  the  appropriate  answers  and  generate  solutions.  To 
encourage  retention,  instructors  should  review  or  summarize  major 
teaching  points  to  remind  the  students  why  they  are  important.  To 
elicit  more  active  learning  in  students,  instructors  can: 

•  Ask  students  to  summarize. 

•  Involve  students  by  providing  obviously  wrong  information,  which 
challenges  them  to  think  about  and  discuss  their  knowledge  and 
beliefs. 

•  Divide  students  into  small  groups  to  get  them  involved  in 
discussions. 

•  Ask  questions  randomly  during  lectures. 

Student  involvement  increases  when  instructors  relate  directly  relevant 
“war  stories’*  or  anecdotes  and  explain  their  relevance  clearly. 

When  instructors  tell  students  what  they  are  expected  to  learn  and 
demonstrate  the  steps  needed  to  accomplish  a  task,  students  learn 
better.  This  “direct  instruction”  takes  the  students  through  the 
learning  steps  systematically,  helping  them  to  see  both  the  purpose 
and  the  result  of  each  activity.  Direct  instruction  is  particularly 
effective  in  teaching  basic  skills  and  in  helping  experienced  higher 
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Reading  List: 


21.  Practice 


Finding: 


Comments: 


ability  students  master  complex  materials  and  develop  individual 
study  skills. 


Rosenshinc.  B  .  &  Stevens.  R.  (1986).  Teaching  functions  In  M  C.  Wittrock  (Ed  ), 
Handbitok  of  research  on  leaching  (3rd  ed.)  New  York  Macmillan  Publishing 
Companv 

Rothkopf.  E.  Z  (I98I1  A  macroscopic  model  of  instruction  and  purposeful 
learning  Instructional  Science.  10(2).  105-122. 

Sullivan.  H  .  &  Higgins.  N  (1983).  Teaching  for  competence  New  York:  Teachers 
College.  Columbia  University. 

Thiagarjan.  S  (1985).  25  Ways  to  improve  anv  lecture.  Performance  and  Instruction 
Journal.  24(1(1).  22-24. 

Walberg.  H  J  (1984.  May).  Improving  the  productivity  of  America  s  schools 
Educational  Leadership.  41(8).  19-36. 


Opportunity  to  practice  lesson-related  tasks  promotes  learning  new 
skills. 


Students  learn  more  by  doing  than  by  watching  or  listening.  They 
should  have  opportunities  to  practice  the  steps  of  any  procedures  they 
are  learning.  They  should  practice  the  new  behaviors  in  a  variety  of 
situations  that  represent  conditions  in  which  what  is  being  learned 
is  used. 

Instructors  need  to  provide  practice  opportunities  since  practice 
improves  performance.  Separate  each  repetition  of  identical  or  similar 
drills  with  other  drill  activities. 

Emphasize  the  key  points  during  practice  to  increase  the  likelihood 
that  students  address  and  recall  these  key  points.  Explicit  feedback 
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about  performance  helps  students  identify  and  correct  performance 
difficulties. 

The  amount  of  practice  required  to  perform  a  task  correctly  usually 
increases  with  the  complexity  of  the  task.  In  very  complex  tasks, 
however,  components  of  the  tasks  need  to  be  learned  and  practiced 
first  and  then  combined  later.  For  example,  in  air-traffic  control 
training  students  spend  much  time  practicing  the  entire  task. 
Acceptable  performance  levels  may  not  be  attained  by  the  end  of  the 
training  program. 

Sometimes,  using  the  wrong  learning  strategy  prevents  learning 
with  practice.  In  seemingly  simple  tasks  such  as  memorizing  strings 
of  digits,  students  can  practice  for  hours  without  improving  their 
performance  unless  they  are  shown  how  to  use  grouping  and  coding 
schemes  to  help  them  learn. 


Reading  List:  Geuinger.  M.  (1984)  Individual  differences  in  time  needed  for  learning:  A  review  of 
the  literature  Educational  Psychologist.  19(1).  15-29. 

Schneider.  W.  (1985).  Training  high-performance  skills:  Fallacies  and  guidelines 
Human  Factors.  27(3).  285-300. 

Sullivan.  H..  &  Higgins.  N.  (1983).  Teaching  for  competence.  New  York:  Teachers 
College.  Columbia  University 


22.  Peer  Teaching 


Finding:  Peer  “teachers”  and  their  students  receive  higher  grades  on  tests  and 

develop  more  positive  attitudes  toward  the  courses  in  which  peer 
teaching  has  occurred. 


Comments:  Peer  interaction  improves  the  academic  performance  and  attitudes 

of  the  students  who  receive  tutoring  and  those  who  provide  it. 
Instructors  can  supplement  regular  classroom  and  laboratory  teaching 
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with  peer  teaching.  It  helps  slower  and  underachieving  students  to 
learn  and  succeed  in  school.  The  peer  teachers  benefit  from  preparing 
and  giving  lessons  to  other  students  because  they  learn  more  about 
the  lessons  they  prepare  and  present. 

Peer  teaching  can  take  a  variety  of  forms  such  as: 

•  Teacher  assistants  leading  discussion  groups,  seminars,  or  tutorial 
groups. 

•  Senior  students  assisting  individual  students  (called  the  proctor 
model). 

•  Student  learning  groups  that  are  instructor-less  or  self-directed. 

Student  coaching  usually  raises  achievement  test  scores.  The  effects 
are  greatest  in  longer  courses.  Short  test-taking  oriented  courses  show 
the  least  improvement  as  a  result  of  coaching  methods. 

Students  bring  a  lot  of  invaluable  life  experiences  into  the  classroom, 
which  should  be  acknowledged  and  used.  Students  can  learn  much 
from  dialogue  with  respected  peers. 


Peer  proctoring  in  a  Navy  technical  school  substantially  reduced 
attrition  in  contrast  to  attrition  in  the  standard  course. 

Senior  students  assisted  individual  students  in  learning  using  the 
Personalized  System  of  Instruction  (PSI)  which  is  the  proctor 
model.  They  made  sure  that  learners  did  assignments,  took  tests, 
and  restudied  materials  when  necessary.  The  PSI  was  developed  to 
assist  in  college  teaching,  but  has  been  used  in  many  adult  training 
situations.  PSI  courses  are  mastery  oriented,  student  proctored. 
self-paced,  and  use  printed  study  guides  to  guide  students'  studying, 
and  occasional  lectures  and  discussion  to  stimulate  and  motivate  the 
students.  Reviews  of  numerous  studies  comparing  PSI  taught  courses 
with  those  taught  by  standard  classroom  procedures  find  substantially 
better  achievement  for  PSI  students  on  various  tests  given  during 
and  after  the  course,  even  larger  superiority  on  tests  given  weeks  or 
months  later,  and  better  attitudes  toward  the  course. 

In  a  Navy  technical  training  program  in  Propulsion  Engineering, 
implementation  of  PSI  resulted  in  a  substantial  reduction  in  attrition 
as  shown  in  the  figure  below.  The  graph  shows  the  number  of 
students  dropped  from  the  course  for  several  months  in  1973  when 
it  was  fixed  length  and  primarily  lecture-based  and  several  months  in 
1974  when  PSI  was  implemented. 


57 


Typical  behaviors  of  students  involved  with  peer  tutoring: 

(1)  Almost  always  more  motivated;  seem  to  have  increased  attention 
span  and  engage  in  more  spontaneous  conversation; 

(2)  More  active  and  involved  in  their  classes; 

(3)  Participation  in  learning  increases  and  the  nature  of  their 
discourse  becomes  more  analytic  and  problem-solution  oriented  — 
which  results  in  a  large  increase  in  vocabulary; 

(4)  Increased  responsibility  for  learning  (topics,  pacing,  and 
commitment). 


Reading  List;  Assinder.  W.  (1991).  Peer  teaching,  peer  learning:  One  model.  ELT  Journal.  45(3). 
218-229. 

Bangert-Drowns.  R.L..  Kulik.  J.A.,  &  Kulik.  C.C.  (1983.  Winter).  Effects  of 
coaching  programs  on  achievement  test  performance.  Review  of  Educational 
Research,  53(4).  571-585. 

Cohen.  P.  A  .  Kulik.  J.  A  .  &  Kulik.  C.  C.  (1982.  Summer).  Education  outcomes 
of  tutoring:  A  meta-analysis  of  findings.  American  Educational  Research  Journal. 
19(2),  237-248. 

Goldschmid.  B  .  &  Goldschmid.  M.  L.  (1976).  -Peer  teaching  in  higher  education:  A 
review.  Higher  Education.  5(1),  9-33. 

Kulik.  J.  A..  Bangert-Drowns.  R.  L..  &  Kulik.  C.  C.  (1984).  Effectiveness  of 
coaching  for  aptitude  tests.  Psychological  Bulletin.  95(2).  179-188 
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McSlichael.  J  S  .  Brock.  J.  F  .  &  Delong.  J  (1976)  Job-relevant  Navy  training 
and  Keller's  Personalized  System  of  Instruction:  Reduced  attrition  Journal  of 
Personalized  Instruction.  1(1).  41-44. 

Slavin.  R  E  (1983).  Cooperative  learning  New  York:  Longman.  Inc 


23.  Listening  Skills 


Finding:  Students  can  be  taught  to  improve  their  listening  skills  and  this  will 

increase  their  achievement. 


Comments:  Students  learn  more  from  the  act  of  listening  than  from  any  other 

perceptual  act.  While  it  is  widely  assumed  that  learners  receive  so 
much  daily  practice  in  listening  that  formal  training  is  unnecessary, 
this  assumption  is  incorrect  for  many  adult  learners;  many  major 
corporations,  thus,  have  had  to  institute  listening  skills  programs.  Too 
many  adults  do  not  perceive  much  spoken  stimuli  around  them,  some 
fail  to  process  it  accurately  and  many  more  fail  to  remember  what 
they  have  heard:  too  many  listeners  have  bad  listening  habits. 

An  individual  can  be  taught  to  listen!  Many  learners  will  achieve 
more  after  formal  listening  training.  As  a  result,  many  colleges, 
industries  and  government  groups  have  established  classes  to  improve 
learning  skills.  Learning  to  listen  is  difficult.  Retraining  students  to 
listen  requires  cognitive  restructuring  for  learning  new  concepts  and 
constant  and  appropriate  application  of  good  listening  techniques. 
These  students  must  acquire  a  positive  attitude  about  learning  to 
listen  and  generate  determination  toward  this  end  to  improve  their 
listening  habits. 

There  appear  to  be  seven  productive  listening  techniques  which  an 
instructor  can  teach  to  students: 

•  Generate  interest  in  the  speakers  topic.  Look  at  the  speaker. 

Study  the  body  language  —  does  it  support  what  the  speaker  says? 
Show  the  speaker  your  interest  by  your  body  language  (reflective 
listening). 

•  Adapt  to  the  speakers  appearance  and  delivery. 
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•  Adjust  to  distractions.  Your  biases  can  restrict  your  perception. 
Abstain  from  faking  attention  and  pretending  to  listen. 

•  Listen  for  the  concepts  and  major  ideas  instead  of  facts. 

•  Listen  to  difficult  expository  material  carefully.  Then,  interpret, 
evaluate  and  remember  the  most  significant  major  points  (or  facts) 
by  writing  them  down  or  memorizing  them. 

•  Listen  to  the  entire  message  before  judging  and  refuting.  Avoid 
interruptions  or  asking  questions  which  disrupt  the  speakers 
presentation  unless  the  speaker  encourages  interaction  —  or 
begins  to  ramble.  Ask  necessary  questions  after  the  idea  has  been 
presented. 

These  skills  can  be  learned  as  a  result  of  participating  in  exercises 
intended  to  refine  these  skills.  After  listening  to  a  prerecorded 
tape  focusing  upon  one  or  more  techniques,  checklists  are  often 
useful  to  help  the  learner  focus  on  what  they  didn't  perceive  or 
their  misconceptions.  Instructors  can  function  as  facilitators  of 
student  learning,  in  addition  to  their  traditional  role  as  presenters  of 
ready-made  information,  by  teaching  them  to  listen. 


Reading  List:  Burley-Alien.  M.  (1982)  Listening  The  forgotten  skill.  NY:  John  Wiley  &  Sons. 

Montgomery.  R  L.  (1981).  Listening  made  easy.  New  York:  AMACON 

Murphy.  K.  J.  (1987).  Effective  listening  Your  key  to  career  success.  New  York: 
Bantam  Books. 

Stewart.  J.  (Ed  )  (1986)  Bridges  not  walls:  A  book  about  interpersonal 
communications  (4th  ed.)  NY:  Random  House. 

Wolff.  F.  I..  Marsnik.  N.  C..  Tacey.  W.  S..  &  Nichols.  R  G  (1983)  Perceptive 
listening  New  York:  Holt.  Rinehart  &  Winston. 


24.  Homework  Assignments 


Finding:  Student  performance  improves  significantly  when  instructors  regularly 

give  out-of-class  assignments,  make  sure  they  are  completed,  and  give 
explicit  feedback  about  the  adequacy  of  the  completed  assignment. 
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Comments:  Students  learn  significantly  more  from  their  assignments  when 

instructors  write  comments  and  grades  on  student  papers. 
Furthermore,  students  in  courses  that  require  out-of-class  assignments 
learn  more  than  do  students  in  courses  without  such  assignments. 

The  time  students  spend  on  relevant  out-of-class  assignments  benefits 
them  as  much  as  in-class  learning  time. 

Instructors  can  use  out-of-class  assignments  to  increase  practice, 
which  can  be  especially  helpful  for  lower  achievers.  Low  ability 
students  who  spend  several  hours  on  out-of-class  assignments  often 
obtain  grades  as  high  as  students  with  greater  ability  w-ho  do  no  extra 
assignments.  These  assignments  boost  student  achievement  because 
they  increase  total  study  time,  which  influences  how  much  a  student 
learns.  This  can  be  helpful  for  all  students,  but  may  be  especially 
important  for  those  who  are  lower  achievers. 

Students  are  more  willing  to  do  assignments  they  consider  useful. 

To  benefit  student  learning,  instructors  can  give  the  same  care  to 
preparing  the  out-of-class  assignments  as  they  give  to  classroom 
instruction,  treat  them  as  an  integral  part  of  instruction,  evaluate 
them,  and  count  them  as  part  of  the  course  requirements. 


Reading  List:  Graue.  M.  E  .  Weinstein.  T..  &  Walberg.  H.  J.  (1983).  School-based  home 
instruction  and  learning:  A  quantitative  synthesis.  Journal  of  Educational 
Research.  76(6).  351-360. 

Paschal.  R.  A..  Weinstein.  T.,  &  Walberg.  H.  J.  (1984).  The  effects  of  homework 
on  learning:  A  quantitative  synthesis.  Journal  of  Educational  Research.  78(2), 
97-104. 


Walberc.  H.  J.  (1983).  Educational  achievement  and  economic.  Daedulus.  112(2). 
1-28. 

Walberg.  H.  J.  (1983).  Synthesis  of  research  on  teaching.  In  M.  C.  Wittrock 
(Ed.).  Third  handbook  of  research  on  teaching.  Washington,  D.C.:  American 
Educational  Research  Association. 

Walberg.  H.  J.  (1984).  Improving  the  productivity  of  American’s  schools. 
Educational  Leadership .  41(8).  19-36. 

Yeshurun.  S.  (1973).  A  reinforcement  study  on  service  courses  in  mathematics 
and  statistics.  International  Journal  of  Mathematics  Education  in  Science  and 
Technology.  4(2).  127-132. 
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25.  Testing  Student  Learning 


Finding:  Frequent,  systematic  testing  and  assessing  of  student  progress  informs 

students  about  their  learning  and  instructors  and  managers  about 
strengths  and  weaknesses  in  student  learning  and  the  instruction. 


Comments:  Instructors  test  students  and  assess  their  work  to  learn  what  students 

already  know  and  what  they  need  to  learn.  They  use  various  means, 
including  observing  laboratory  exercise  performance,  giving  oral 
quizzes  and  tests,  assigning  homework,  asking  questions  in  the 
classroom,  and  giving  comprehensive  performance  tests. 

Student  errors  on  tests  and  in  class  alert  instructors  to  learning 
problems  that  need  to  be  corrected.  Student  motivation  and 
achievement  improve  when  instructors  provide  prompt  feedback  on 
their  performance  and  assignments.  “Teaching  to  the  test"  provides  a 
good  alignment  between  objectives,  content  and  a  test  as  long  as  the 
test  examines  the  performance  desired.  The  test  should  not  focus  on 
criterion-referenced  items  but  upon  primary  application  requirements 
found  in-the-world. 

In  technical  training,  assessment  should  be  as  job-like  as  possible. 

This  means  emphasizing  hands-on  performance  tests,  limiting 
pencil-and-paper  tests  to  safety  and  knowledge  critical  for  job 
performance,  and  stressing  open-book  testing  in  which  students 
use  manuals  and  other  references  normally  available  on-the-job. 
Frequently  tested  students  outperform  less  frequently  tested  ones  in 
the  classroom. 

Students  generally  take  either  knowledge  or  performance  tests. 
Knowledge  tests  help  instructors  find  out  if  the  students  have  learned 
information  important  for  safety  and  knowledge  important  for 
performance. 

Performance  tests  enable  instructors  to  determine  student 
competence,  and  identify  student  and  instruction  problems.  The 
instructors  biggest  concern  with  testing  is  to  identify  what  the  students 
do  not  know  .  Performance  difficulties  often  indicate  gaps  in  student 
knowledge,  and  their  explanations  of  their  actions  or  answers  to 
questions  can  confirm  an  instructor's  inference. 


62 


Wha!  Works  in  Adul!  Instruction 


389 


Reading  List:  Bangert-Drowns.  R  L  ei  a!  (1986.  April)  Effects  of  frequent  classroom  testing 
Paper  presented  at  the  Annual  Meeting  of  the  American  Educational  Research 
Association.  San  Francisco.  CA. 

Ellis.  J  A  .  &  Wulfeck  11.  W.  H.  (1986).  Criterion-referenced  measurement 
in  militarv  technical  training.  In  J.  A.  Ellis  (Ed  ).  Miluan  contributions  to 
instructional  technology  (pp.  60-82).  New  York:  Praeger  Publishers 

Ellis.  J  A..  Kmrk.  F  G..  Taylor.  B.  E..  &  McDonald.  B.  A.  (1987  March) 

The  course  evaluation  system.  San  Diego,  CA:  Navy  Personnel  Research  and 
Development  Center 

Roid.  G..  &  Haladvna.  T.  (1982).  A  technology  for  test  item  writing.  New  York: 
Academic  Press. 

Samson.  G  E  .  Graue.  M  E..  Weinstein.  T..  &  Walberg.  H.  J  (1984).  Academic 
and  occupational  performance:  a  quantitative  synthesis.  American  Educational 
Research  Journal.  21(2).  311-321. 

Shepard.  L  A  (1991).  Psychometricians'  beliefs  about  learning.  Educational 
Researcher.  20(6).  2-16. 


26.  Giving  Feedback  to  Students 


Finding:  Students  who  receive  constructive  feedback  about  the  accuracy  and 

adequacy  of  their  performance  become  more  interested  in  the  class  and 
learn  more. 


Comments:  Giving  constructive  feedback  to  students  about  the  adequacy 

and  accuracy  of  their  actions  is  an  effective  way  for  instructors  to 
aid  student  learning.  Timely  comments  about  their  performance 
provide  important  recognition  of  their  effort  and  help  correct 
errors.  Feedback  for  retrieval  and  application  of  specific  knowledge 
stimulates  the  correction  of  erroneous  responses  where  its  reception  is 
encouraged. 

No  one  method  is  best  for  providing  feedback  to  students,  but 
instructors  can  follow  some  useful  general  rules.  Whether  or  not  an 
answer  is  correct,  the  feedback  should  be  prompt  and  provide  useful 
information.  Even  after  a  correct  answer,  feedback  emphasizing  the 
method  used  to  get  the  correct  answer  reinforces  the  solution  and,  if 
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Reading  List: 


other  students  are  onlookers,  they  can  understand  why  the  answer  is 
correct. 

Instructors  should  give  nonspecific  praise  and  criticism  infrequently 
and.  even  then,  base  it  on  the  quality  of  student  performance.  It  is 
better  to  explain  correct  or  incorrect  performance  than  to  give  only 
the  correct  answer  or  to  judge  the  student  performance.  Feedback 
should  routinely  tell  students  when  they  are  incorrect,  but  should 
focus  on  the  content  and  explain  how  to  reach  the  correct  answer. 
Critical  feedback,  written  or  spoken,  should  be  given  in  private  and 
not  in  front  of  the  class. 

By  giving  constructive,  timely  feedback,  instructors  can  reinforce  and 
help  students  develop  positive  self-esteem  as  well  as  improve  their 
performance.  Students  who  believe  they  can  succeed  are  usually 
more  successful  than  those  who  are  less  sure  of  their  ability.  Usually, 
students  who  believe  they  can  succeed  are  more  active  learners,  work 
independently,  cooperate  with  other  students,  and  achieve  more. 

Recent  research  suggests  that  feedbback  can  be  overdone.  A 
committee  of  The  National  Research  Council  reported  that  too  much 
feedback  from  instructors  may  be  bad  for  long-term  performance. 
While  feedback  is  an  essential  part  of  learning,  it  occurs  naturally 
when  a  learner  practices.  Augmented  feedback  from  instructors 
—  such  as  telling  a  trainee  what  he  or  she  is  doing  wrong  on  each 
attempt  —  can  be  excessive.  Reducing  augmented  feedback  to 
perhaps  once  in  every  five  trials  may  even  improve  performance. 

Feedback  within  programmed  materials  also  seems  to  positively  affect 
learning.  Mediated  intentional  feedback  for  retrieval  and  application 
of  specific  knowledge  appears  to  stimulate  the  correction  of  erroneous 
responses  in  situations  where  its  mindful  reception  is  encouraged. 


Bangert-Drowns.  R  L.,  Kulik.  C.  C..  Kulik.  J.  A.,  &  Morgan.  M.  T.  (1991) 

The  instructional  effect  of  feedback  in  test-like  events.  Review  of  Educational 
Research.  61(2).  213-238. 

Brophv.  J.  (1981).  Teacher  praise:  A  functional  analysis.  Review  of  Educational 
Research.  51(1).  5-32. 

Brophv.  J  .  &  Good.  T.  L.  (1986).  Teacher  behavior  and  student  achievement  In 
M.  C.  Wittrock  (Ed  ).  Handbook  of  research  on  teaching  (3rd  ed..  pp.  328-375) 
New  York:  Macmillan  Publishing  Company. 

Carlsen.  W.  S.  (1991).  Questioning  in  classrooms:  A  sociolinguistic  perspective 
Review  of  Educational  Research .  61(2).  157-178. 
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Crooks.  T  J  (1988).  The  impact  of  classroom  evaluation  practices  on  students 
Ri -wen  of  Educational  Research.  58(4).  43k— 481 

Lysakowski.  R  S..  3c  Walberg  H  J  (1981).  Classroom  reinforcement  and  learning 
A  quantitative  synthesis  Journal  of  Educational  Research.  75(2).  69-77. 

Orlich.  D  (1985).  Teaching  strategies,  a  guide  to  better  instruction  Lexington.  MA: 
Heath  and  Company . 

Swets.  J.  3c  Bjork.  R  (1990.  March)  Enhancing  human  performance:  An  evaluation 
of  "New  Age"  techniques  considered  by  the  U.S.  army  Psychological  Science. 
1(2).  85-96. 
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FINDINGS  ABOUT  THE  EXPLOITATION  OF  INSTRUCTIONAL  TECHNOLOGY 

Decisions  concerning  the  use  of  instructional  technology  need  »o  be  made 
as  a  part  of  the  curriculum  design  process.  A  decision  to  use  an  instructor, 
computer,  distance  learning  program,  and/or  other  media  represents 
major  cost,  instructional  design  and  time-to-implementation  commitments. 
Individual  student-oriented  simulations,  gaming,  and  virtual  reality  decisions 
permit  the  integration  of  many  skills  in  a  manner  that  is  impossible  with 
more  traditional  group-oriented  media.  Distributed  instruction  may  permit 
a  significant  reduction  in  training  costs  compared  to  alternative  media. 
Instructional  technology  can  be  exploited  in  many  ways  to  improve  the 
learning  process. 
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27.  Cost  Effectiveness 


Finding:  Consistent  and  credible  evaluations  of  cost-effectiveness  must  justify 

any  plans  to  substitute  alternative  instructional  programs  for  those 
now  in  use. 


Comments:  Because  of  the  rapid  decline  in  the  cost  of  computer-based  and 

communications  technologies  during  the  last  decade,  they  are 
attractive  alternatives  to  the  conventional  lecture  method  of  providing 
instruction  for  mans  types  of  programs.  Interactive  videodisk 
instruction,  for  example,  has  been  found  to  significantly  increase 
learning  and  decrease  learning  time  in  both  industrial  training 
and  in  higher  education.  The  interactive  video  training  was  also 
less  expensive  than  traditional  instructional  methods.  These  cost 
savings  are  found  only  if  the  teacher-pupil  ratio  is  increased.  If  the 
traditional  components  and  processes  (T/P  ratio,  grading  procedures, 
classroom  sizes,  fixed  length  of  class  periods)  are  unchanged  when 
these  media-based  systems  are  introduced,  the  new  technology  merely 
increases  the  already  high  cost  of  instruction.  To  offset  or  justify 
the  cost  of  the  technology,  benefits  should  be  demonstrable.  For 
example,  instructor  productivity  or  the  number  of  students  graduated 
in  a  time  period  should  increase,  student  performance  should  improve 
substantially,  or  administrative  burdens  over  the  life  of  the  system 
should  be  reduced.  Such  changes  require  good  management  planning 
as  well  as  changes  to  the  instructional  program. 

The  decision  to  implement  a  particular  instructional  program,  course 
or  device  or  to  change  an  existing  one  rests  upon  identifying  all 
the  costs  of  all  the  alternatives  such  as  the  cost  of  research  and 
development,  all  personnel  costs  in  development,  the  development 
as  well  as  delivery  costs  of  all  versions  of  the  equipment,  the  cost 
of  running  the  implementation  for  the  life  of  the  system  including 
operation  and  maintenance.  Then,  if  the  instructional  systems 
demonstrate  about  the  same  effectiveness,  the  one  that  costs 
less  might  be  preferred.  Substantial,  demonstrated  differences  in 
instructional  effectiveness  might  justify  choosing  a  more  costly 
system.  Both  cost  and  effectiveness  must  be  considered  explicitly  in 
analysis  conducted  to  enable  selection  among  alternative  instructional 
programs,  courses,  or  devices  to  fulfil  a  specific  need. 
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Cost-effectiveness  of  instruction  can  be  improved  by  the  use  of 
instructional  technology  even  if  it  often  entails  additional  personnel 
costs  to  select/evaluate,  design,  and  revise,  these  materials.  The 
important  related  questions  involve  a  discussion  of  whether  or  not 
to  use  computers,  to  the  more  relevant  question:  how  to  assign  and 
implement  the  appropriate  interactive  technology  in  specific  classes. 


Reading  List:  Fletcher.  J  C  .  Hawley.  D  E  .  &  Piele.  P  K.  (1990)  Costs,  effects,  and  utility  of 
microcomj  uter  assisted  instruction  in  the  classroom  American  Educational 
Research  Journal.  27(4).  783-806 

Fletcher.  J  D  (1900).  Effectiveness  and  cost  of  interactive  videodisc  instruction  in 
defense  training  and  education.  IDA  Paper  P-2372.  Alexandria.  V'A.  Institute  for 
Defense  Analysis. 

Knapp.  M  1  .  Orlansky .  J.  (1983.  November)  A  cost  element  structure  for  defense 
iraimtiQ  i IDA  Paper  P-1709)  Alexandria.  VA  Institute  for  Defense  Analysis 

Orlansky.  J  .  Siring.  J.  (1981.  Second  Quarter)  Computer-based  instruction  in 
military  training  Defense  Management  Journal.  18(2).  46-54. 

Ntemtec,  R  P  .  &  Walbcre.  H  J.  (1987).  Comparative  effects  of  computer-assisted 
instruction:  A  synthesis  of  reviews  Journal  of  Educational  Computing  Research. 
3.  19-3“. 


28.  Programming  Instructional  Presentation 


Finding:  Instructional  materials  can  be  organized  systematically,  can  be 

presented  by  various  media,  and  are  more  effective  than  conventional 
classroom  instruction. 


Comments:  Structuring  instructional  materials  by  dividing  them  into  learnable 

segments,  determining  their  presentation  order,  and  requiring 
students  to  pass  tests  to  demonstrate  their  comprehension  before 
allowing  them  to  go  to  new  material  works  about  as  well  as 
conventional  methods.  Structured  instructional  materials,  or 
programmed  instruction,  also  provides  students  with  an  opportunity  for 
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Reading  List: 


self-paced  study,  which  can  save  considerable  training  time,  and  can 
be  distributed  to  remote  locations  as  alternatives  to  lectures 

Sequences  of  instruction  are  designed  to  require  an  active  response 
from  students  before  new  information  is  presented.  Students 
get  immediate  feedback  telling  them  whether  the  response  was 
correct  Sometimes  branching  enables  students  to  omit  material 
they  already  know  .  If  students  make  errors  they  may  be  required 
to  study  segments  again.  This  method  of  organizing  and  presenting 
information  can  be  used  to  deliver  instruction  on  various  media 
such  as  computers,  workbooks,  or  lectures.  Many  computer-aided 
instruction  (CAI)  programs  are  examples  of  structured  instruction; 
others  use  simulation  or  gaming  techniques. 

Students  who  progress  through  the  materials  at  their  own  rate 
complete  the  materials  in  about  one-third  less  time  than  do  students 
who  attend  conventional  courses. 

While  some  students  will  learn  a  particular  task  regardless  of  the 
delivery  medium,  others  take  advantage  of  a  medium's  specific 
characteristics  to  help  construct  knowledge.  For  example,  the 
serial  processing  of  linguistic  and  pictorial  information  in  books  is 
influenced  by  the  medium.  The  information  provided  by  television 
is  influenced  by  the  simultaneous  presentation  of  its  symbol  systems 
and  the  information  coded  in  it.  Computers  influence  learning  with 
their  ability  to  dynamically  represent  formal  constructs  and  institute 
procedural  relationships  under  the  learner's  control. 


Kozma.  R.  B.  (1991)  Learning  with  media.  Review  of  Educational  Research ,  61(2). 
179-211. 

Kulik.  C  C..  Schwalb.  B.  J..  &  Kulik.  J.  A  (1982).  Programmed  instruction  in 

secondary  education:  A  meta-analysis  of  evaluation  findings.  Journal  of  EducanonaJ 
Research .  75(3).  133-138. 

Orlansky.  J.,  &  String.  J.  (1981.  Second  Quarter).  Computer-based  instruction  for 
military  training.  Defense  Management  Journal,  46-54. 

Tobias.  S.  (1973)  Sequence,  familiarity,  and  attribute  by  treatment  interactions  in 
programmed  instruction.  Journal  of  Educational  Psychology,  64,  133-141. 
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29.  Programmed  Instruction  Workbooks 


Finding:  Programmed  training  materials  reduce  training  time  about  a  third 

compared  with  standard  instruction  without  affecting  student 
achievement. 


Comments:  Aristotle  said.  "We  learn  when  what  we  do  has  reinforcing 

consequences."  Reinforcement  is  basic  to  programmed  instruction 
(PI).  PI  presents  information  in  a  structured  manner  congruent  with 
current  learning  theories.  PI  emphasizes  graded  difficulty,  corrective 
feedback  and  self  pacing.  The  PI  procedure  is  often  equated  with  the 
general  concept  of  "instructional  design." 

The  basic  steps  in  developed  PI  materials  are:  (1)  Perform  a  task  and 
learner  analysis:  (2)  develop  behavioral  objectives;  (3)  write  and  edit 
materials  in  accord  with  current  learning  theories:  and  (4)  Perform 
empirical  testing  and  revision  until  the  objective  "conditions”  and 
"standards"  are  met  —  or  until  the  materials  are  rejected. 

Programmed  instruction  usually  results  in  equivalent  learning  in  from 
one-half  to  two-thirds  the  time  required  to  lecture  the  equivalent 
material,  or.  as  learning  time  is  increased,  the  standards  tend  to  be 
exceeded. 

PI  tends  to  be  cost-effective,  even  given  the  large  front-end 
development  effort,  because  instructors  can  monitor  more  students 
so  instructor  costs  may  be  reduced.  Where  students  are  either  at 
multiple  sites  or  where  they  may  be  available  to  start  instruction  over 
a  period  time.  PI  permits  additional  time  savings  by  permitting  them 
to  start  instruction  "on  demand”  because  of  its  self-pacing  character. 

The  use  of  individualized  "programmed  materials"  allows  an 
instructor  to  help  specific  learners  with  their  unique  problems  —  thus 
potentially  increasing  the  quality  of  instruction. 


Reading  List:  Bullock.  D.  H.  (1978)  Programmed  instruction.  Englewood  Cliffs.  NJ:  Educational 
Technology  Publications 

Goldstein.  I  L.  (1986).  Training  in  organizations  Seeds  assessment,  development, 
and  evaluation  Belmont.  CA:  Wadsworth. 
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Skinner.  B  F  (19,Nbi  Procram  instruction  revisited  Phi  Delia  Kappan.  68(2i. 
103-110 

Rutkaus.  M  A  (19N^)  Remember  programmed  instruction'.’  Educational 
Technology  .  27(10).  4b— IS 


30.  Computer-Based  Instruction 


Finding:  Computer-based  instruction  teaches  students  the  same  content  as  well 

as  or  better  than  a  regular  classroom  situation,  usually  in  less  time. 


Comments:  A  review  of  nearly  200  studies  comparing  computer-based  instruction 

(CBI)  with  conventional  elementary,  secondary,  and  college 
classroom  instruction  found  that  computer-based  instruction 
raised  student  achievement  significantly,  gave  students  a  better 
appreciation  of  technology,  improved  student  attitudes  toward  schools 
and  teaching  and  helped  teachers  manage  instructional  time.  A 
review  of  40  studies  comparing  military  classroom  instruction  with 
computer-based  instruction  found  that  CBI  student  performance 
achievement  improved  in  15  cases,  remained  the  same  in  24  cases 
and  became  worse  in  1.  In  addition,  students  completed  the  CBI 
lessons  in  about  30%  less  time  than  that  allotted  for  the  conventional 
courses.  This  finding  may  be  important  where  students  are  paid  and 
training  time  needs  to  be  as  brief  as  possible.  This  evidence  tends  to 
verify  the  suitability  of  computer-based  training  in  the  military. 

Th  se  effectiveness  and  efficiency  gains  did  not  result  simply  from 
using  computers  in  instruction  but  from  imposing  a  systems  approach 
for  design  on  the  courses  and  allowing  students  to  progress  at  their 
own  learning  rates.  In  military  courses  where  course  materials  and 
tests  already  address  training  objectives  derived  from  job-task 
analysis,  gains  in  student  performance  would  not  be  expected, 
although  time  savings  compared  with  the  length  of  conventional 
courses  would  be  expected.  Therefore,  careful  planning  is  necessary 
before  deciding  to  use  computer-based  instruction  in  each  situation 
and.  only  if  cost-effectiveness  evaluation  justifies  its  use.  should  it  be 
adopted. 
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Sequences  of  instruction  are  designed  to  require  an  active  response 
from  students  before  new  information  is  presented.  Students 
get  immediate  feedback  telling  them  whether  the  response  was 
correct.  Sometimes  branching  enables  students  to  omit  material 
the>  already  know.  !f  students  make  errors  they  may  be  required 
to  study  segments  again.  This  method  of  organizing  and  presenting 
information  can  be  used  to  deliver  instruction  on  various  media 
such  as  computers,  workbooks,  or  lectures.  Many  computer-aided 
instruction  (CAI)  programs  are  examples  of  structured  instruction; 
others  use  simulation  or  gaming  techniques.  Computers  have 
the  capability  of  creating  dynamic,  symbolic  representations  of 
nonconcrcte.  formal  constructs  which  are  frequently  missing  in  the 
mental  models  of  novice  learners.  Learners  can  manipulate  these 
representations  on  computers  to  work  out  differences  between  their 
incomplete,  inaccurate  mental  models  and  the  formal  principles 
presented  via  the  computer. 

Students  who  progress  through  the  materials  at  their  own  rate 
complete  the  materials  in  about  one-third  less  time  than  do  students 
who  attend  conventional  courses.  Better  vet.  studies  of  students 
using  computers  for  instruction,  show  that  students  learn  at  least  as 
well,  or  better,  when  paired  at  a  computer  as  they  do  by  working 
by  themselves  on  a  computer.  Students  who  use  CAI  for  drill  and 
practice  in  pairs  perform  better  than  students  who  use  the  same 
program  individually.  Learner  attitudes  toward  their  class-mates 
also  improves  when  they  are  paired.  Cooperative  learning  promotes 
positive  feelings  of  personal  worth  and  positive  attitudes  toward  the 
course  content;  low  ability  students  who  study  mathematics  in  pairs 
usually  have  reduced  anxiety  toward  mathematics.  The  pairing  of 
learners  makes  CAI  relatively  cost  effective. 

Learners  prefer  having  an  instructor  present  the  instruction,  partly 
because  they  have  learned  to  learn  in  a  iecture  situation.  They  prefer 
sitting  in  a  classroom  with  a  human  being  who  can  listen  and  respond 
rather  than  sitting  in  a  media  carrel  with  a  computer  terminal  or  a 
slide-tape  program.  Student  attitudes  toward  the  content,  however, 
do  not  vary  much  in  either  situation. 

On-line  CBI  is  as  effective  as  classroom-based  lectures  when 
computer  communication  (electronic  conferencing/electronic  mail) 
between  faculty  and  students  is  possible.  The  Virtual  Classroom 
project  involving  several  courses  at  the  New  Jersey  Institute  of 
Technology  and  Upsala  College  found  that  students  were  more 
actively  involved  in  the  computer  classes  and  attitudes  were  increased 
with  no  measurable  differences  in  learning. 
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Example:  In  military,  training  courses,  computer-based  instruction  has  been 

found  to  be  at  least  as  effective  as  standard  lecture  courses  and 
students  complete  them  substantially  faster.  The  following  figure, 
taken  from  Orlansky  and  String  (1979).  provides  a  graphic  summary 
ot  a  review  of  40  research  studies  comparing  the  effectiveness  of 
computer-based  and  standard  training  in  the  Navy.  Army,  and  Air 
Force.  Fifteen  studies  reported  higher  achievement  for  students 
in  the  computer-based  courses,  and  36  reported  that  students  in 
computer-based  courses  finished  in  less  time. 


Reading  List:  Fletcher.  J.  D  .  Hawley.  D.  E..  &  Piele.  P.  K.  (1990).  Costs,  effects,  and  utility  of 
microcomputer  assisted  instruction  in  the  classroom.  American  Educational 
Research  Journal.  27(4).  783-806. 

Kozma.  R  B.  (1991).  Learning  with  media.  Review  of  Educational  Research.  61(2). 
179-211. 


Mevarech.  Z  R..  Silber.  O  ,  &  Fine.  D.  (1991).  Learning  with  computers  in  small 
groups:  Cognitive  and  affective  outcomes.  Journal  of  Educational  Computing 
Research.  7(2).  233-243. 

Orlansky.  J..  &  String.  J.  (1979.  April).  Cost-effectiveness  of  computer-based 

instruction  in  military  training  (IDA  Paper  P-1375).  Alexandria.  VA.  Institute  for 
Defense  Analyses. 

Rogoff.  B  .  &  Lave.  J.  (1984).  Everyday  cognition:  Its  development  in  social  context. 
Cambridge.  MA:  Harvard  University  Press. 

Whyte.  M.  M..  Kmrk.  F  G..  Casey.  R.  J..  &  Willard,  M.  L.  (1991).  Individualistic 
versus  paired  cooperative  computer-assisted  instruction;  Matching  instructional 
method  with  cognitive  style.  Journal  of  Educational  Technology  Svstems.  19(4). 
299-312 
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3).  H>perte\t-Based  Instruction 


Finding:  The  use  of  random  access  computer  technologies.  like  hypertext, 

makes  practicable  new  forms  of  nonlinear  and  multidimensional 
learning  and  instruction  better  suited  to  conveying  complex  content. 


Comments:  Traditional  methods  of  instruction  as  lectures,  textbooks,  and 

vide  tapes  rely  on  linear  presentation  strategies.  For  subject  matter 
which  is  well  structured  and  relatively  low  in  objective  taxonomies 
these  linear  methods  are  adequate.  For  complex  content  which  is 
ill-structurca  the  unidimensionality  of  linear  structures  is  not  efficient. 
Instructional  techniques  which  allow  a  degree  of  random  access  to  a 
variety  of  media  can  be  very  efficient. 

The  need  for  hypertext-based  instruction  has  been  demonstrated 
in  such  diverse  areas  as  literary  comprehension  and  interpretation, 
biomedical  cognition,  history,  and  military  strategy.  In  literary 
study  the  mere  chronicling  of  events  is  not  sufficient;  a  literary 
work  usually  contains  many  themes  and  symbolic  structures  which 
lend  themselves  to  study  in  diverse  w-avs.  be  relevant  at  irregular 
intervals  and  allow  the  study  of  intricate  patterns  of  various 
themes.  Tran:fer-related  objectives  are  often  made  possible  by  this 
technology.  This  random-access  form  of  instruction  may  alert  students 
to  the  hazards  of  oversimplification  and  to  prepare  them  to  deal  with 
complexity  ac.^^s  content  domains. 

The  role  of  the  student-user  is  one  of  active  participant  and  explorer 
within  the  “data-base  "  as  opposed  to  relatively  inactive  listener. 

Instructional  design  of  hypertext-based  materials  varies  from 
traditional  instructional  development  activity.  Many  hypertexts  are 
based  on  small,  relatively  uncomplicated  case-studies  which  can  be 
thoroughly  explored  by  the  user.  By  overlaying  more  points  of  view 
on  the  same  content  materials  permits  an  incremental  buildup  of 
a  picture  of  interrelations  among  the  thematic  perspectives.  Many 
instructional  units  will  not  have  a  “solution”  but  will  retain  an 
openness  of  data  interpretation  —  right  and  wrong  is  often  hard  to 
determine  in  the  real  world.  Two  of  the  major  problems  of  hypertext 
instructional  systems  involve  the  labyrinth  of  content  connections  and 
the  need  to  prestore  their  links. 
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Reading  List:  B.irvulou.  L  W  (!9S~i  Instability  of  graded  structure:  Implications  for  the  nature  of 
concepts  In  l  Scissor  (Ed  ).  Concepts  and  conceptual  development  Ecological 
tout  intellectual  factors  in  categorization  (pp  101-140).  Cambridge  Cambridge 
University  Press 

Feltovieh.  P.  J  .  Spiro.  R  J  .  &  Coulson.  R  L.  (1989)  The  nature  of  conceptual 
understanding  in  biomedicine:  The  deep  structure  of  complex  ideas  and  the 
development  of  misconceptions.  In  D.  Evans  <fc  V.  Patel  (Eds  ).  The  cognitnc 
science s  in  medicine  (pp  113-172).  Cambridge.  MA:  MIT  Press. 

Jonassen.  D.  H  (Ed  )  (1990)  Designing  hypermedia  for  learning.  New  York: 
Springer- Verlag 

Spiro.  R.  J..  &  Jehng.  J.  (1990)  Cognition,  education,  and  multimedia  Exploring 
ideas  in  high  technology.  Hillsdale.  NJ:  Lawrence  Erlbaum  Associates. 


32.  Visual  Representations  for  Learners:  Television/Interactive  VideoA'ideoteletraining 
( VTT  i 


Finding:  A  lecture  delivered  using  instructional  television  is  about  as  effective 

as  a  live  instructor.  Television  works  better  than  an  instructor-based 
system  when  realistic  visuals  are  required  to  master  the  content. 


Comments:  Interactive  video  instruction  (IV)  is  an  effective  form  of  instruction. 

The  effects  of  using  interactive  video  instruction  is  similar  to  that  of 
computer-assisted  instruction. 

Visual  attention  is  positively  influenced  by  “formal  features"  such  as 
different  types  of  voices.  laughing,  sound  effects,  music,  and  visual 
movement  (high  physical  activity,  animation,  cuts,  zooms,  pans). 

The  combined  use  of  visual  and  auditory  symbols  result  in  more 
viewer/learner  recall  than  visual-only  or  audio-only  presentations; 
when  presented  together  each  source  provides  additional, 
complementary  information.  In  video  presentations,  the  linguistic 
information  should  be  presented  simultaneously  with,  or  just 
following,  the  visual  information.  Video  viewers  seem  to  use  their 
prior  domain  knowledge  to  process  information  at  the  presentation 
speed  and  supplement  information  they  miss  or  do  not  understand:  if 
the  topic  is  unfamiliar  they  will  have  little  supplementing  information 
in  long-term  memory  and  the  pace  of  the  presentation  may  easily 
exceed  their  capacity  to  process  the  presentation  (and  learning 
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will  tail  to  occur).  Pacing  should  reflect  the  skill  level  of  the  video 
presentations  intended  audience. 

There  are  differences  in  student  achievement  between  conventional 
grouped  instruction  and  IV.  The  effects  of  learning  from  interactive 
videodisk-based  training  systems  is  also  a  little  different  than  that 
resulting  from  noninteractive  videotape-based  systems.  It  appears 
that  when  comparing  IV  instruction  with  noninteractive  instruction, 
or  conventional  instruction,  that  achievement  may  improve  by  as 
much  as  15  percentage  points,  based  on  24  military  studies,  while 
14  studies  in  higher  education  raised  the  achievement  by  25  points. 
Many  of  the  studies  favoring  group  instruction  seem  to  be  explained 
by  factors  such  as  decisions  made  by  the  teacher  relative  to  the 
amount  of  practice,  the  extent  and  kind  of  feedback,  and  the  nature 
of  remediation  procedures. 

External,  program  controlled  interactive  video  instruction  appears  to 
be  more  effective  than  learner-controlled  IV.  IV  is  best  accomplished 
when  it  is  structured  and  provides  guidance  as  opposed  to  being 
primarily  under  the  control  (pacing,  sequence  of  instruction)  of  the 
students. 

Live  video  presentations,  video  taped  lessons,  video  distance 
learning  and  interactive  video  courses  often  provide  time  and  cost 
effective  alternatives  to  traditional  instructional  procedures.  Video 
presentations,  like  all  other  presentations,  are  delivery  systems  for 
communicating  course  content  and  are  not  ends  in  themselves. 

Both  live  television  courses  and  video  taped  lessons  provide  effective 
alternatives  to  conventional  instruction  for  most  learners  and  most 
course  content.  In  many  cases  where  the  instruction  is  provided 
by  video  distance  learning  techniques,  the  method  is  very  cost 
effective:  it  eliminates  most  travel  and  per  diem  costs  (a  major 
cost  consideration)  while  increasing  instructional  hardware  and 
communication  costs  (moderate  cost  items).  In  addition,  this  form 
of  communication  allows  course  managers,  instructional  developers 
or  other  instructors  to  easily  monitor  the  quality  and  uniformity 
of  lessons  before  (if  they  are  recorded)  and  during  the  broadcast 
presentation. 

Interactive  video  courses  have  shown  they  will  often  increase  student 
learning  and/or  decrease  learning  time  and/or  decrease  training  costs 
as  compared  to  traditional  training  methods.  Interactive  videodisk 
instruction  is  more  effective  the  more  the  interactive  features  of  the 
medium  are  used.  The  method  is  equally  effective  for  knowledge  and 
performance  outcomes. 
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Videoteleiraining  (\'TT)  systems  are  extensively  used  in  higher 
education,  industrial  training  and  in  the  military.  VTT  provides  a  low 
cost  alternative  for  providing  remote  instruction.  VTT  ma>  involve 
the  use  of  videotapes,  interactive  videodisks  or  audiographics.  The 
biggest  problem  in  using  VTT  involves  obtaining  adequate  audio 
feedback  from  the  students  to  the  instructor.  VTT,  like  most  other 
instructional  television-based  instructional  systems  works  at  least  as 
well  as  conventional  training  methods  for  most  courses. 


Reading  List:  Bailey.  S  S  .  Sheppc.  M.  L  .  Hodak.  G.  W  .  Kruger.  R.  L..  &  Smith.  R.  F. 

( 1 989).  Video  teletrainmg  and  video  teleconferencing  A  review  of  the  literature. 
Technical  Report  89-036:  Naval  Training  Systems  Center. 

Carl.  D  L  (1991)  Electronic  distance  learning  Positives  outweigh  negatives. 
THE  Journal.  18(10).  67-70. 

Fletcher.  J  D  ( 1990)  Effectiveness  and  cost  of  interactive  videodisc  instruction  in 
defense  training  and  education.  IDA  Paper  P-2372  Alexandria.  N'T  Institute  for 
Detense  Analyses. 

Kozmu.  R  B.  (1991).  Learning  with  media  Review  of  Educational  Research.  61(2). 
179-211. 


McNeil.  B  J..  &.  Nelson.  K  R.  (1991).  Meia-analvsis  of  interactive  video 

instruction:  A  10  year  review  of  achievement  effects.  Journal  of  Computer-Based 
Instruction.  18(1).  1-6 

Pugh.  H  L..  Parchman.  S  W  .  &  Simpson.  H.  (1991).  Field  sursey  of  videoteletratnmg 
systems  in  puhhc  education,  industry,  and  the  military  (NPRDC  TR-91-7).  San 
Diego.  CA  Navy  Personnel  Research  and  Development  Center. 

Salomon.  G..  &  Gardner.  H.  (1986).  The  computer  as  educator:  Lessons  from 
television  research  Educational  Researcher.  15(1).  13-19. 


33.  Simulations  and  Simulation  Games 


Finding:  Simulations  and  simulation  games  provide  effective  instruction  in 

complex  skills  as  substitutes  for  instruction  on  equipment  in  the  actual 
performance  environment. 
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Instructional  simulations  abstract  portions  of  a  real  environment 
in  order  to  provide  instruction  that  otherwise  would  require  an 
actual,  "hands-on"  performance  situation.  The  real  situation  ma\ 
be  dangerous  (e.g..  practicing  how  to  react  to  nuclear  powerplant 
malfunctions),  distant  (e.g.,  transporting  the  learners  to  the  moon 
to  experience  the  effects  of  its  gravity),  expensive  (e.g..  using  a 
flight  simulator  to  practice  aircraft  landings  without  the  expense 
of  operating  an  aircraft),  or  politically  sensitive  (e.g..  learning 
the  communications  and  other  problems  involved  when  a  local 
community,  and  government  agencies  confront  industrial  pollution). 
Computer-based  devices  are  often  used  to  provide  the  simulation. 
They  range  in  realism,  from  high  fidelity  (e.g.,  regarding  the  visual 
and  motion  systems  of  the  actual  tasks)  to  (lower  fidelity)  abstract 
symbolic  representations  of  actual  tasks  or  partial  tasks.  A  simulation 
game  can  also  vary  in  realism,  but  adds  an  additional  set  of  rules  to 
introduce  competition  and  determines  a  winner  or  winners. 

Simulation  devices  provide  learners  with  situated  learning  that 
exercises  the  cognitive  and  physical  activities  of  the  real  task. 

Whether  the  physical  actions  need  to  exactly  duplicate  those  of  the 
real  task  is  an  important  issue  for  design  because  that  requires  high 
physical  fidelity  at  considerable  cost.  Research  has  shown  that  lower 
fidelity  devices  can  provide  learning  that  translates  into  good  job-task 
performance.  Recent  research  emphasizes  a  psychological,  conceptual 
fidelity  as  the  primary  dimension  to  consider  rather  than  physical 
fidelity. 

As  an  example,  simulations  provide  students  with  self-contained 
"micro-worlds",  such  as  representing  on  a  computer  screen  a 
frictionless  world  where  the  laws  of  Newtonian  physics  are  more 
apparent;  one  in  which  the  learner  can  experiment  and  quickly  see 
results.  Learners  can  test  abstract  concepts  and  experiment  with 
scientific  processes  that  are  not  feasible  or  are  too  dangerous  for 
actual  classroom  work  (such  as  the  operation  of  a  ship's  propulsion 
system).  Simulations  are  also  effective  tools  for  training  police 
how  to  manage  social  or  other  sensitive  problems.  By  playing  the 
role  of  world  leaders  or  citizens  in  other  countries,  for  example, 
trainees  can  engage  in  high  level  critical  thinking,  gain  a  better 
understanding  of  political  affairs,  and  appreciate  different  perspectives 
on  sensitive  issues. 

Instructional  simulations  and  games  provide  situated,  authentic 
forms  of  practice,  feedback  about  errors,  depict  how  a  device  or 
system  works,  and  can  motivate  learning  while  avoiding  physical 
danger  and  temporal  constraints.  In  addition,  simulations  can  use 
strategics  effective  for  supporting  student  learning  about  the  task.  For 
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example,  rather  than  simply  replicating  the  timing  of  real  events,  in  a 
simulation  timing  can  he  slowed  for  the  new  learner,  and  speeded-up 
later.  Explanations  can  be  given  at  decision  points  to  support 
learning,  and  tc  correct  errors.  Normally  invisible  events  or  processes 
can  be  made  visible  to  promote  understanding,  such  as  showing  the 
flow  of  electrons  in  devices.  Therefore,  at  times,  simulations  may  be 
preferable  to  instruction  in  the  actual  performance  situation. 

Simulations  intended  to  instruct  novices  may  need  to  be  designed 
quite  differently  than  those  intended  to  provide  practice  or  retraining 
for  moderately  competent  students.  Novices  need  extensive  guidance 
and  precise  corrective  feedback  to  reduce  their  errors,  while  trained 
students  may  hone  their  skills  ana  broaden  their  knowledge  of  new 
and  unusual  situations/cues/events  that  may  be  encountered  in  the 
"real  world.”  New  learners  need  simplified  examples  of  problems  to 
facilitate  their  learning.  The  effectiveness  of  a  simulation  results  from 
the  instructional  methods  incorporated  into  the  device  that  support 
student  learning  rather  than  from  any  simple  physical  or  functional 
similarity  to  an  actual  device.  Thus,  the  design  decisions  are  based, 
for  example,  on  ways  to  isolate,  or  discriminate  cues,  and  ways  to 
provide  time-compressed  practice  for  skill  development. 

An  instructional  simulation  game  is  like  a  simulation  but  has  an 
additional  set  of  rules  (often  mathematical)  to  identify  a  winner  or 
winners.  These  games  which  have  individuals  compete  against  each 
other  are  motivating  for  many  learners  and  are  good  at  teaching 
objectives  involving  procedures  and  decision-making  or  psveh^-motor 
development.  Adventure  games,  as  opposed  to  more  instructional^- 
oriented  games,  seem  to  have  limited  application  in  the  classroom 
beyond  overcoming  a  users  fear  of  computers  and  helping  them  to 
understand  how  to  operate  the  machine. 


Reading  List:  Alessi.  S  M.  (1988).  Fidelity  in  the  design  of  instructional  simulations.  Journal  of 
Computer-Based  Instruction.  15(2).  40-47. 

Blaiwes.  A.  S..  &  Regan.  J.  J.  (1986)  Training  devices:  Concepts  and  progress.  In 
J.  A.  Ellis  (Ed  ).  Military  contributions  to  instructional  technology  (pp.  83-170). 
New  York:  Praeger  Publishers. 

Grabe.  M..  &  Dosmann.  M.  (1988).  The  potential  of  adventure  games  for  the 
development  of  reading  and  study  skills.  Journal  of  Computer-Based  Instruction. 
15(2).  72-77. 

Office  of  Technology  Assessment  (1988)  Power  on!  New  tools  for  teaching  and 
learning  Washington.  D  C  :  U  S.  Government  Printing  Office. 
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34.  Distributed  or-  Distance  Instruction 


Finding:  Students  not  at  formal  schools  can  interact  with  instructors  through 

modern  communications  technology  such  as  networked  computers  or 
through  an  instructional  television  network. 


Comments:  Through  instructional  electronics  networks,  apprentices,  alone  or 

in  small  groups,  can  learn  skills  and  knowledge  where  they  will  use 
them.  A  telephone  computer  network  controls  audio  or  electronic 
exchanges  between  students  and  instructors,  while  satellite,  cable,  or 
cassettes  deliver  video  if  needed.  Participants  can  work  on  problems 
peculiar  to  their  own  situation  when  their  scheduled  work  allows. 
Vacations  are  possible:  for  example,  participants  can  delay  the 
interactions  by  storing  questions,  answers,  and  comments  until  they 
have  time  to  address  them. 

Microcomputers  can  also  serve  as  terminals  to  remote  data  banks  and 
network  members.  Through  telephone  connections  and  a  centralized 
message  workspace,  learners  can  ask  questions  or  propose  solutions 
to  other  members  sharing  the  network. 

Several  sound  educa'iortal  benefits  result  from  distributed  instruct. on. 

•  Enables  beginning  apprentices  to  observe  interchanges  between 
more  experienced  apprentices  and  instructors  and  to  develop  their 
skill  in  approaching  problems  gradually. 

•  Reaches  learners  where  and  when  the  training  is  '""''ed. 

•  Shifts  more  responsibility  for  acquiring  the  skill  from  the  trainer  to 
the  learner. 

•  Individualizes  studving  and  increases  student  interaction  with 
materials. 

•  Uses  the  learner's  and  trainer’s  time  more  productively. 

•  Saves  travel  time,  cost,  and  time  away  from  the  job. 


Reading  List:  Bergman.  R  E  (1981).  Technology  and  'raining:  The  shape  of  tomorrow's  seminar. 
Performance  and  Instruction  .  20(9),  17-20. 

L  i.  J  A  .  Riel.  M..  Miyake.  N..  &  Cohen,  M.  (1987).  Education  on  the  electronic 
frontier.  Teleapf  'entices  in  globally  distributed  educational  contexts.  Contemporary 
Educrn^i  al  Psychology.  12(3).  254-260 

Newman.  D  (1987)  Local  and  long  distance  com;  iter  networking  for  science 
classrooms  Educational  Technology.  27(6).  20-23. 
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35.  Job  Performance  Aids 


Finding:  Job  performance  aids  are  effective  training  devices  for  skill 

deselopment. 


Comments:  Job  aids  differ  irom  training  aids  in  that  training  aids  are  documents 

or  devices  designed  to  encourage  learning  of  particular  skills 
or  knowledge,  whereas  job  aids  are  designed  to  assist  in  the 
performance  of  work  activities  in  the  work  environment.  But.  job 
aids  can  help  learners  become  proficient  at  a  task,  fulfilling  an 
instructional  role 

Job  aids  serve  in  many  roles  including  maintenance  and/or 
troubleshooting  tasks,  cueing,  analogs,  and  examples.  Job  aids  that 
serve  as  procedural  models  typically  provide  step-by-step  directions 
for  accomplishing  an  objective,  whereas  those  whose  purpose  is  to 
provide  performance  cues  are  designed  to  provide  signals  to  act  in  a 
specific  way  without  complete  step-by-step  directions.  Job  aids  serving 
an  associative  role  are  often  designed  to  enable  the  user  to  look  up 
data  relating  to  an  existing  piece  of  information. 

The  effectiveness  of  job  aids  have  been  demonstrated  in  many 
situations  and  tasks  conditions.  In  one  case  a  job  aid  was  developed 
for  a  sophisticated  radar  system  and  was  to  be  used  by  low-aptitude 
novice  maintenance  technicians.  A  comparison  was  made  to 
determine  the  time  saved  by  technicians  who  used  the  job  aid  versus 
those  who  used  the  conventional  manuals  which  already  existed  for 
the  system.  The  novice  technicians  using  the  job  aids  were  capable 
of  reducing  the  time  required  to  isolate  and  correct  a  malfunction  by 
as  much  as  50%  over  the  time  required  by  highly  trained  technicians 
using  conventional  procedures. 

To  design  a  job  aid  the  following  procedure  should  be  considered:  (1) 
Identify  the  tasks  elements  for  which  job  aids  should  be  considered. 
(2)  determine  the  job  aid  functional  characteristics  which  will  enhance 
the  performance  of  each  task  element,  (3)  specify  the  physical 
design  characteristics  required  of  the  aids  to  carry  out  the  necessary 
functions,  and  (4)  evaluate,  modify,  and  update  the  resulting  aids. 

The  evaluation  step  is  a  critical  element  in  the  design  process  as  it 
provides  feedback  to  the  designer  as  to  the  effectiveness  of  the  job 
aid.  A  set  of  criteria  that  can  be  used  for  evaluating  job  aids  which 
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arc  essentially  applicable  to  any  information  presentation  strategy  are: 
Comprehensiveness,  degree  of  prescription,  reiiability/validity .  user 
opinion,  and  cost. 


Example:  The  Air  Force  has  developed  the  “fully  procedurahzed"  job 

performance  aids  or  JPAs.  A  salient  feature  of  fully  procedurahzed 
JPAs  is  that  they  are  entirely  job-oriented  and  provide  illustrated 
step-by-step  instructions  for  the  performance  of  linear  tasks,  that 
is.  tasks  whose  sequence  of  steps  never  vary,  such  as  cleaning, 
lubrication,  inspection,  removal,  and  installation,  as  well  as  branching 
diagnostic  routines  (such  as  check-out  and  troubleshooting  activities). 


Recc -r  devotion  jr-ve  ossempiy 


Disassemble  elevation  drive  assembly 

1  Remove  two  cogjioi  connector  nuts  (8) 

2  Remove  s*  screws  U) 

3  Depress  two  coomol  connectors  tT)  mto  end  piote  :6' 

4  Remove  end  plate  (2) 

5  Remove  s.x  Katneod  screws  <5? 

6  Remove  closure  seoi  (4) 

■’  Remove  v-sea;  (3) 

Q  Remove  *ovr  screw  ()5)  Remove  puord  d6) 

9  Disconnect  two  COQXiOl  connectors  (IT) 
iC  Remove  *Ouf  screws  00)  ond  two  clomps  (9 
i:  Remove  four  screws  (10 

!2  Remove  si*  screws  U4)  end  si*  washers  Remove  rotary 
,omt  support  U3)  and  hold  rotary  tomt  <I2'  with  one  hand 
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A  task  analytic  process  must  be  complete  prior  to  JPA  development 
sc'  that  the  JPA  can  be  precisely  specified.  This  involves  a  large 
matrix  in  which  every  possible  task  performed  on  each  piece  of 
equipment  is  identified.  Furthermore,  the  training  as  well  as  the 
experience  level  of  a  document's  user  is  carefully  defined,  and  each 
task  identified  in  the  matrix  is  assigned  for  coverage  either  via  a 
JPA  or  via  training  Every  hardware  component  whose  failure  must 
be  identified  is  listed.  In  addition  to  troubleshooting  logic,  the  final 
product  includes  step-b\-step  procedures  for  access  and  adjustment 
tasks.  All  procedures  are  necessarily  accompanied  by  illustrations. 
The  figure  on  page  408  shows  an  example  of  a  JPA  frame. 


Reading  List:  Joyce.  R  P  .  Chenzoff.  A  P  .  Mulligan.  J  F  .  it.  Mallor>.  W  J  <19o).  Fully 
proceduralized  job  performance  aids  Yol  II.  Handbook  for  JPA  developers 
(AFHRL-TR-~3-43|II)|  ( AD‘7?5-705)  Wright-Patterson  Air  Force  Base.  OH:  U  S 
Air  Force  Human  Resources  Laboratory 

Smillie.  R  J  1 1985 ).  Design  strategies  for  job  performance  aids  In  T  M  Duffy  i k 
R  Waller  (Eds .1.  Designing  usable  text  (pp.  9b9-977)  New  York:  Academic 

Swezey.  R  W.  (198")  Design  of  job  aids  and  procedure  venting  In  G  Salvendy 
(Ed.).  Human  factors  handbook  New  York:  Wiles  it  Sons 


36.  Adopting  Training  Innovations 


Finding:  Managers  and  training  developers  can  effect  the  rate  at  which  the 

schools  and  instructors  adopt  and  use  newly  developed  training 
materials  and  programs. 


Comments:  From  the  time  an  instructional  development  project  is  first  conceived, 

developers  and  schools  should  consider  the  strategies  to  use  to 
encourage  the  potential  users  to  adopt  the  new  materials.  Too  often, 
materials  are  adopted  only  where  they  were  developed  —  that  is.  the 
"not  developed  here”  syndrome.  One  way  of  overcoming  this  attitude 
is  to  involve  all  potential  users  in  the  analysis  and  design  phases  of 
innovative  courses. 
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Using  an  effective  person  as  an  agent  to  manage  change  is  a 
critical  factor  in  diffusion.  The  agent  studies  the  potential  adopting 
organization  and  systematically  shows  that  (1)  the  innovation 
has  obvious  advantages  over  the  existing  process,  materials, 
or  equipment.  (2)  it  is  compatible  with  the  existing  system.  (3) 
significant  research  and/or  evaluations  reveal  the  innovation's 
advantage.  (4)  there  is  a  rational  sequence  for  its  adoption  and 
application.  (5)  it  addresses  an  identified  need  of  the  potential  user. 
(6)  the  innovation  will  be  used  for  a  long  time,  and  (7)  the  staff  can 
acquire  the  skills  needed  to  adopt  the  innovation. 

The  manager  must  concentrate  on  the  potential  users  and  their 
needs  rather  than  on  the  material  or  the  innovation,  must  know  as 
much  about  the  situation  as  the  potential  user,  be  ready  to  tailor  the 
innovation  to  the  user's  needs,  and  explain  the  innovation  to  the 
potential  users. 

For  any  instructional  technology,  as  for  computer-based  technology, 
to  have  an  impact  on  the  learning  process,  teachers  must  be 
comfortable  with  computers,  seeing  them  as  tools  that  enhance 
rather  than  interfere  with  their  daily  teaching.  For  this  to  happen, 
instructors  need  special  training.  Unfortunately,  the  vast  majority 
of  today  s  teachers  have  had  little  or  no  training  on  how  to  applv 
computers  or  other  individualized  teaching  technologies  in  their 
instruction.  Recent  reports,  summarized  by  the  Office  of  Technology 
Assessment,  suggest  that  only  about  one-third  of  all  instructors  have 
had  even  10  hours  of  computer  training. 


Margulies.  N.  ei  al.  11973).  Organizational  change  Techniques  and  applications 
Glenview  IL  Scott.  Foresman 

Nelson.  M  .  &  Sieher.  S.  D  (I976j  Innovations  in  urban  secondary  schools  School 
Rene*.  840.  213-231 

Office  of  Technology  Assessment  (1988)  Power  on!  Sew  tools  for  teaching  and 
learning.  Washington.  D  C  U  S.  Government  Printing  Office 

Rogers.  E.  (1983)  Diffusion  of  innovations  (2nd  ed  ).  New  York:  Free  Press 
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37.  Designing  Effective  Illustrations  and  Graphs 


Proposition:  Diagrams,  graphs,  photos,  and  illustrations  can  improve  student 

learning. 


Comments:  Illustrations  enhance  text  instruction  by  helping  the  students  perceive 

and  remember  the  instruction.  Illustrations  should  be  as  simple  as 
possible  to  reduce  potential  confusion  with  irrelevant  details.  For  that 
reason,  line  drawings  are  often  more  effective  than  complex  drawings 
or  p  otographs.  Color  illustrations  are  needed  only  when  color  itself 
is  important  to  cue  what  is  being  learned. 

Pictures  or  other  illustrations  not  directly  related  to  the  presentation 
are  unnecessary.  Such  materials  are  often  more  distracting  than 
helpful.  Highlighted  or  labeled  information  can  aid  learning,  but  be 
careful  to  avoid  confusing  clutter.  Several  illustrations  to  show  the 
various  switches  or  components  relevant  to  the  current  instruction  are 
more  comprehensible  than  one  cluttered  one. 

The  use  of  color  may  encourage  students  to  examine  the  materials. 

In  these  cases,  the  advantages  outweigh  the  potential  confusion. 
Animation,  use  of  many  visuals  changing  at  a  rapid  pace  may 
increase  student  attention  to  a  presentation;  this  technique  may  be 
used  for  training  where  the  students  may  have  little  interest  in  the 
course  or  content. 


Example:  Effective  Illustration:  Two  illustrations  are  contrasted  in  the  following 

figure.  The  valve  on  the  left  is  from  a  Navy  rate  training  manual.  As 
a  depiction  for  explaining  how  such  valves  operate,  it  is  confusing 
and  cluttered.  It  needs  to  be  simpler  and  show  only  those  parts  most 
intimately  involved  in  the  process  of  reducing  pressure  from  one 
level  to  another.  Irrelevant  labeling  and  construction  detail  were 
removed  on  the  valve  at  the  right  to  show  the  pans  most  important 
for  explaining  how  the  valve  works. 
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Once  it  is  decided  that  an  illustration  is  required  to  support  learning, 
a  simple  illustration  depicting  just  the  relevant  detail  is  preferable 
(Fleming,  1987).  The  design  of  a  display  is  determined  by  attention  to 
the  perceptual  limitations  of  students,  limits  on  their  ability  to  process 
information,  and  knowing  about  what  students  know  and  understand. 
Since  these  characteristics  are  primarily  qualitative,  tryouts  with 
typical  students  are  useful  for  validating  design  decisions  and  revising 
illustrations. 


Reading  List:  Fleming.  M  L.  (1987).  Displays  and  communication.  In  R.  M.  Gagne  (Ed.). 

instructional  technology:  Foundations.  Hillsdale.  NJ:  Lawrence  Erlbaum 
Associates 

Levie.  W.  H..  &  Lentz.  R.  (1982).  Effects  of  text  illustrations:  A  review  of 

research.  Educational  Communication  and  Technology  Journal.  30(3).  195-232. 

Levin.  J.  R..  Anglin.  G.  J..  &  Carney,  R.  N.  (1987).  On  empirically  validating 
functions  of  pictures  in  prose.  In  D.  M.  Willows  &  H.  A.  Houghton  (Eds  ),  The 
psychology  of  illustration:  /.  Basic  research.  New  York:  Springer- Veriag. 

Wulfeck.  W  H..  Chang.  F  R..  &  Montague.  W.  E.  (1986)  Document  and  display 
design  In  T  Sticht.  F  Chang  &  S.  Wood  (Eds  ).  Advances  in  reading/ language 
research  Volume  4.  1986,  Cognitive  science  and  human  resources  management 
( pp  183-195).  Greenwich,  Conn:  JAI  Press. 
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FINDINGS  ABOUT  THE  EVALUATION  OF  INSTRUCTION 

Following  the  analysis  of  the  training  requirements,  the  design  of  the 
instruction,  and  the  production  of  the  training  program,  it  is  desirable  to 
determine  if  the  training  program  works  and  what  the  students  have  learned. 
The  use  of  formative  evaluation  will  permit  the  instructional  developers 
to  improve  the  quality  of  the  current,  and  future  materials.  Summative 
evaluation  will  ensure  that  there  is  a  good  correlation  between  the  intended 
training  outcomes  and  the  real  ones.  The  assessment  will  also  permit  the 
identification  of  instructors  who  need  additional  training. 
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38.  Criterion  Referenced  Testing 


Finding:  Testing  needs  to  be  geared  closely  to  the  goals  of  a  training  program. 


Comments:  Testing  during  and  after  instruction  is  used  to  indicate  student 

progress,  determine  what  students  find  difficult,  and  tailor  individual 
assignments  to  overcome  the  difficulties.  The  testing  therefore,  is 
focused  on  performance  requirements  which  are  derived  from  analysis 
of  the  work  trained  individuals  are  expected  to  do.  Various  means 
of  testing  are  used  including  laboratory  exercise  performance,  oral 
and  written  quizzes  and  tests,  out-of-class  assignments,  classroom 
questions,  and  comprehensive  performance  tests. 

Assessment  needs  to  be  as  job-like  as  possible.  This  performance- 
based  assessment  should  make  the  elements  of  the  test  as  much 
like  the  elements  of  the  criterion-based  objectives  as  efficiency  will 
permit.  Performance  tests  should  be  hands-on  and  pencil-and-paper 
tests  of  know  ledge  should  be  restricted  to  safety  and  knowledge 
critical  for  job  performance.  If  workers  use  manuals  and  books 
to  find  the  information  needed  to  carry  out  a  task  on-the-job. 
open-book  testing  should  be  used. 

Well  designed,  performance-oriented  tests  inform  students  about  job 
requirements  and  guide  their  learning.  Frequently  tested  students 
outperform  less  frequently  tested  ones.  Students  generally  take  two 
kinds  of  tests:  knowledge  tests  and  performance  tests.  Knowledge 
tests  help  instructors  find  out  if  the  students  have  learned  information 
important  for  safety,  and  knowledge  important  for  performance. 
Performance  tests  indicate  student  competence  and  provide  information 
about  both  student  and  instruction  inadequacies.  Errors  that  students 
make  on  tests  and  in  class  identify  learning  problems  that  need  to  be 
corrected.  Instructors  need  this  information  to  provide  prompt  feedback 
to  students  on  their  performance  and  assignments  and  to  help  correct 
any  difficulties  they  may  have. 


Reading  List:  Baker.  E.  L..  &  O'Neil  Jr.  H  F.  (1987).  Assessing  instructional  outcomes.  In 

R  M.  Gagne  (Ed  ).  Instructional  technology:  Foundations  Hillsdale.  NJ:  Lawrence 
Eribaum  Associates 
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Bunycn-Dro«.ns.  R  L.  (19Xh.  April)  Effects  of  frequent  classroom  testing 
America*,  Educational  Research 

Ellis.  J  A  .  &  Wulfeck  II.  W  H  (1986)  Critenon-referenced  measurement  in  military 
technical  training  In  J  A  Ellis  (Ed  ).  Military  contributions  to  instructional 
technology  (pp  60-82).  New  York:  Praeger  Publishers. 

Ellis.  J  A  .  knirk.  F.  G..  Taylor.  B.  E..  &  McDonald.  B.  A.  (1987).  The  course 
evaluation  system  (NPRDC  TR  87-10).  San  Diego:  Navy  Personnel  Research  and 
Development  Center. 

Linn.  R  L  .  Baker.  E  L  .  &  Dunbar.  S.  B.  (1991).  Complex,  performance-based 
assessment.  Expectations  and  validation  criteria.  Educational  Researcher,  20(8). 
15-21 

Roid.  G  .  &  Haladyna.  T.  (1982)  A  technology  for  test  item  writing  New  York 
Academic  Press 

Samson.  G.  E  .  Graue.  M.  E  .  Weinstein.  T..  &  Walberg.  H.  J.  (1984).  Academic 
and  occupational  performance:  a  quantitative  synthesis.  American  Educational 
Research  Journal.  21(2).  311-321 


39.  Course  Evaluation  and  Revision 


Finding:  Tryouts  during  development  of  instructional  materials  help  to  diagnose 

and  repair  inadequacies  in  the  instruction. 


Comments:  Designing  instruction  involves  making  many  decisions  such  as  how  to 

present  information  to  the  students,  judging  student  comprehension, 
and  knowing  when  they  have  learned  enough  to  move  on  to  new 
material.  The  design  and  development  process  involves  numerous 
subjective  decisions,  and  quality  of  the  instruction  depends  on 
the  skill  and  knowledge  of  the  developers.  The  material  may  only 
approximate  the  optimal  product.  Evaluating  and  revising  the 
instruction  to  improve  it  is  an  important  part  of  the  process. 

The  instructional  developer  accomplishes  this  by  trying  out  segments 
of  material  on  a  sample  of  the  target  students.  Ideally,  a  developer 
goes  through  the  material  with  one  student  at  a  time.  During  tryouts. 
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students  might  he  asked  about  the  quantity  and  quality  of  examples 
in  the  instruction,  the  adequacy  of  opportunities  provided  for 
practice,  the  suitability  of  media  selected  for  a  given  training  domain, 
the  compatibility  of  the  reading  grade  level  of  the  materials  and  the 
student  audience,  and  the  time  required  for  the  student  to  complete 
the  instruction  compared  to  allotted  training  time.  The  developer 
then  revises  the  materials  to  address  problems  uncovered  in  tryout 
and  conducts  another  tryout  with  different  students. 

Training  development  rarely  includes  this  evaluation-revision  cycle. 
Tryouts  of  materials  in  nearly  final  form  are  more  common.  At 
this  late  stage,  however,  it  is  difficult  to  diagnose  instructional 
problems  unless  gross  failure  makes  them  apparent.  The  lack  of 
evaluation  during  development  degrades  quality  and  makes  revision 
of  instruction  a  major  undertaking. 

Managers  who  plan  and  allocate  adequate  resources  for  early 
evaluation  make  revision  and  the  instruction  more  effective. 


Reading  List:  Ellis.  J  A  .  kmrk.  F  G..  Taylor,  B.  E..  &  McDonald.  B  A.  (1987).  The  course 

t'l aluatton  system  NPRDC  TR  87-19  San  Diego.  CA:  Navy  Personnel  Research 
and  Development  Center. 

Markle.  S  M  (1967)  Empirical  testing  of  programs.  In  P  C.  Lange  (Ed.). 
Programmed  instruction:  The  sixty-sixth  yearbook  of  the  national  study  of 
education.  Pan  11.  Chicago:  University  of  Chicago  Press. 

Merrill.  M  D  .  Reigeluth.  C  M-.  &  Faust.  G.  W  (1979).  The  instructional  quality 
profile:  A  curriculum  evaluation  and  design  tool.  In  H  F.  O'Neil  Jr  (Ed  ). 
Procedures  for  instructional  systems  development  (pp.  165-202).  New  York:  NY. 
Academic  Press. 

Montague.  W.  E  .  Ellis.  J  A..  4c  Wulfeck  II.  W  H.  (1983).  Instructional  quality 
inventory:  A  formative  evaluation  tool  for  instructional  development  Performance 
and  Instruction  Journal.  22(5).  11-14. 
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40.  Evaluating  and  Supervising  Instructors 


Finding:  Instructors  can  improve  their  performance  if  they  are  given  relevant 

useful  information. 


Comments:  When  training  organizations  have  students  rate  their  instructors. 

instructors  can  use  the  ratings  to  improve  their  teaching.  Evaluation 
studies  show  that  feedback  from  the  ratings  does  improve  instructor 
performance. 

Instructors  who  receive  mid-course  rating  feedback  obtain 
substantially  higher  end-of-course  ratings  than  do  instructors  who  are 
rated  only  at  the  end  of  the  semester.  Ratings  improve  even  more 
when  instructors  discussed  the  mid-course  ratings  with  consultants  or 
received  other  help  in  interpreting  and  reacting  to  the  ratings. 

Focusing  the  mid-course  evaluations  on  the  entire  course  including 
the  content,  facilities,  instructor,  and  the  instructional  materials, 
seems  to  result  in  more  positive  corrections  than  the  more  traditional 
end-of-course  instructor  evaluations. 

Effective  managers  ensure  that  the  instructors  know  the  subject 
matter  and  can  communicate  it.  The  teaching  skills  of  instructors 
who  know  their  subject  matter  can  usually  be  improved  to  a  higher 
level.  Teaching  is  a  skilled  activity  that  takes  time  and  the  proper 
conditions  to  develop.  Developing  a  skilled  teacher  may  take 
years.  The  most  effective  way  to  develop  instructor  teaching  skills 
is  to  provide  adequate  opportunities  to  teach  under  supervised 
conditions  where  the  observer  may  analyze  inadequacies  and  provide 
constructive  feedback. 

Supervision  that  strengthens  instruction  and  improves  instructor 
morale  has  these  elements: 

•  The  supervisor  and  the  instructor  agree  on  the  specific  skills  and 
practices  that  characterize  effective  teaching. 

•  The  supervisor  observes  the  instructor  frequently  to  verify  that  the 
instructor  uses  these  skills  and  practices. 

•  The  supervisor  and  the  instructor  meet  to  discuss  the  supervisor's 
observations. 

•*  The  supervisor  and  instructor  agree  on  areas  for  improvement. 

•  The  supervisor  and  instructor  jointly  develop  a  specific  plan  for 
improvement. 


91 


41s 


W  E  MONTAGUE  and  F  G  KNIRK 


Reading  List: 


Ciscell.  R  E  ( 198”).  Student  ratings  of  instruction:  Change  the  timetable  to 
improve  instruction  Community  College  Review.  15,  34-38 

Cohen.  P  A.  (1481)  Effectiveness  of  student-rating  feedback  for  improving  college 
instruction  A  meta-analvsis  of  findings  Research  in  Higher  Education.  13. 
321-341 

Diamond.  N  A  (2988).  S.G.l.D.  Small  Group  Instructional  Diagnosis.  Tapping 
student  perceptions  of  teaching.  In  E.  Wadsworth  (Ed  ),  A  handbook  for  new 
practitioners  (pp  89-44).  Stillwater,  OK:  New  Forums  Press. 

NlcKeachie.  W.  (1978).  Teaching  rips  A  guidebook  for  the  beginning  college  teacher 
(7th  ed  )  New  York:  Heath. 
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FINDINGS  ABOUT  THE  MANAGEMENT  OF  INSTRUCTION 

Training  managers  must  he  concerned  with  the  management  of  their  human 
and  material  resources.  Classroom  instructors  and  instructional  developers 
must  be  located  and  trained.  Facilities  for  instruction  and  instructional 
materials  must  be  located  or  developed.  Student  records  must  be  generated 
and  maintained.  This  section  examines  the  evidence  about  how  instructional 
resources  can  be  managed. 
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41.  Planning  Change  within  Organizations 


Finding:  Exploiting  communications  and  computer  technology  can  serve  policy 

goals  and  meet  training  needs  within  resource  constraints. 


Comments:  Many  revolutionary  changes  in  communications  and  computer 

technologies  can  be  used  for  instructional  purposes  with,  or 
sometimes,  instead  of  the  teacher,  books  and  manuals,  and 
chalkboards.  Various  technologies  can  deliver  training  ‘bat  can  be 
as  effective  -,r  even  more  effective  than  current  methods.  To  exploit 
the  advantages  of  these  technologies  requires  good  analysis  and 
planning.  The  capabilities  and  effectiveness  benefits  must  be  mapped 
against  needs  and  the  current  costs  of  training  including  training 
time.  Funding  must  be  found  for  research  and  development  and 
evaluation  of  new  systems  designed  to  make  training  more  effective 
and  efficient. 

The  rapid  development  of  new  technologies  seems  to  point  to  the 
inevitability  of  significant  changes  in  the  way  training  is  accomplished. 
At  the  same  time,  the  potential  costs  of  these  changes  requires 
caution  and  a  practical  outlook.  Claims  of  large  benefits  in 
effectiveness  must  be  substantiated  by  concrete,  conclusive  empirical 
evidence.  Decades  of  research  reveal  that  improvements  in 
instructional  achievement  are  usually  not  due  to  the  communications- 
computer  technology  but  to  redesign  of  the  content.  Permitting  each 
student  to  learn  at  his  own  pace  achieved  with  or  without  computers, 
is  an  important  source  of  the  gain. 

New  technology  may  make  possible  the  delivery  of  novel  forms  of 
instruction  where,  when,  and  in  ways  heretofore  impossible,  as  well 
as  provide  delivery  of  fairly  standard  instructional  matter  as  needed 
to  students  not  assigned  to  schoolhouses.  In  any  case,  large  scale 
implementation  of  training  technologies  that  substantially  change  the 
organization  and  presentation  of  training  should  be  undertaken  only 
after  formal  study  of  its  cost  effectiveness. 

A  more  focused  effort  to  substantially  expand  the  use  of  technology 
in  education  and  attain  more  fully  integrated  applications  across  the 
curriculum  requires  new  strategies  and  perhaps  decision  authority 
at  a  level  high  enough  to  insure  c  minimum  of  interfere. ice  by  those 
vested  in  maintaining  the  established  practices  and  procedures.  The 
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!^SiK.  nevertheless  witnessed  a  tremendous  expansion  in  the  use 
of  VCRs  and  computers.  In  identifying  barriers  to  increased  use  of 
technologs .  mans  training  directors  cite  the  lack  of  funds  to  purchase 
equipment  as  a  serious  problem.  When  funds  are  sought,  a  large 
initial  investment  is  often  necessary  to  purchase  the  critical  mass  of 
startup  equipment.  Once  the  initial  equipment  has  been  purchased 
a  Meads  funding  policy  is  vastly  preferable  to  a  hurried  'year  end" 
policy.  A  consistent  research  finding  shows  that  investments  in 
technology  cannot  be  fully  effective  unless  teachers  receive  training 
and  support  in  its  use. 

The  effective  use  of  instructional  technology  requires  an  extensive 
dialogue  among  instructors  and  educational  researchers.  Instructors 
need  to  be  exposed  to.  and  be  part  of.  new  break-throughs  in 
education:  researchers  need  a  healthy  dose  of  classroom  realities. 
Classroom  trials  of  new  materials  is  essential  to  ongoing  development 
and  necessary  to  assess  what  works.  Contributions  from  many 
disciplines  is  required  to  improve  training  practices. 


Reading  List:  B.ill  D  L.  A.  McDiarmid.  G  W  (1988)  Research  on  teacher  learning  Studving 
how  teachers  knowledge  changes  Action  in  Teacher  Education.  10(2 >.  17-;.'' 

ke.iNev  .  G  I  lu84i  Training  and  technology  A  handbook  for  HRD  professionals 
Reading.  MA.  Addison-Weslev  Publishing  Co 

Nugent.  G  C  1  i%").  Innovations  in  telecommunications  In  R  M  Gagne  (Ed  1. 
Instructional  technology  Foundations  (pp  201—282 )  Hillsdale.  NJ  Erlbaum 
Associates 

Orlanskv.  J  .  &.  String.  J  (1981.  Second  Ouaricn  Computer-based  instruction  for 
miluarv  training  Defense  Management  Journal.  46- 5 4 

VValberg.  H  J  (Ids*.  April)  Curricular  efficients  can  be  attained  Sational 
Association  of  Sccondars  School  Principals  Bulletin.  71(4%).  15-21 
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42.  Managing  instructors 


Finding:  Effective  training  management  policies  improve  instructor  training, 

student  performance,  and  training  time  management. 


Comments:  Successful  training  managers  have  an  accurate  conception  of  the 

important  factors  determining  effective  instruction.  They  keep  this 
conception  in  mind  as  they  interact  with  personnel  and  allocate 
funds.  With  instructional  improvement  as  a  constant  theme,  they 
scrutinize  existing  practices  to  assure  that  instructor  activities  and 
procedures  contribute  to  the  quality  of  the  instructional  program. 

They  make  sure  instructors  are  trained  and  participate  actively  in  this 
process.  Effective  managers,  for  example,  provide  instructors  with 
opportunities  to  improve  their  teaching  and  classroom  management 
skills  They  minimize  instructors'  administrative  chores  and  teaching 
interruptions,  monitor  teaching  performance,  and  provide  constructive 
suggestions  for  improvement. 

Effective  managers  actively  support  learning.  They  create  an  orderly 
environment.  verify  that  instructors  have  all  the  necessary 
instructional  materials  and  assistance  they  need,  work  to  raise 
instructor  morale,  and  create  a  climate  of  achievement  by 
encouraging  new  ideas  and  involving  instructors  in  policy  formation. 


Reading  List:  Bird  T  .  Little.  J  W  (1985).  Instructional  leadership  in  eight  secondary  schools 
(Final  Repori  io  the  U  S.  Department  of  Education.  National  Jnstnute  of 
Education).  Boulder.  CO:  Center  for  Action  (ERIC  Document  No  ED  263694) 

Carmne.  D  R  .  Gersten.  R  .  &  Green.  S.  (1982.  December).  The  Principal  as  an 
instructional  leader:  A  second  look  Educational  Leadership.  40(3),  47-50 

Corcoran.  T.  (1985.  May)  Effective  secondary  schools.  In  R.  Kyle  (Ed.).  Reaching 
for  excellence:  An  effective  schools  sourcebook  (pp.  82-85).  Washington.  DC. 

U  S  Government  Printing  Office. 

Morris.  V.  C  .  Crowson.  R.,  Hurwitz.  E  .  &  Porter-Gehrie.  C  (1986)  Principals 
in  action:  The  reality  of  managing  schools.  Cited  in  What  works:  Research 
about  teaching  and  learning  (2nd  ed..  p.  64)  Columbus.  OH:  Charles  E.  Merrill 
Publishing  Co. 

Skinner.  B  F  (1984),  The  shame  of  American  education  American  Psychologist. 
39(9).  947-954 
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43.  Managing  Student  Learning 


Finding:  Performance-oriented  leadership  improves  both  formal  (intentional) 

and  informal  (incidental)  learning. 


Comments:  Managers  and  instructors  are  primarily  concerned  with  formal 

learning  developed  using  systematic  procedures  designed  to  promote 
effective  training.  However,  promoting  informal  or  incidental  learning 
can  also  further  formal  instruction.  To  manage  learning  effectively 
both  in  and  out  of  the  classroom,  training  managers  should: 

•  Assert  convictions  and  philosophies  with  regard  to  the  importance 
of  learning  by  each  individual. 

•  Specify  the  roles  for  managers  and  instructors  in  managing  learning 
and  training. 

•  Specify  the  role  of  individual  students  in  managing  their  learning 

•  Personally  observe  and  evaluate  the  learning  environment  of 
schools  and  their  surroundings  including.  Who  is  doing  what, 
when,  where,  and  why.  and  how  these  actions  match  their  stated 
philosophy  and  objectives?  How'  does  the  physical  learning 
environment  affect  learning?  What  is  happening  in  the  school  that 
should  not  be  happening? 


Example:  Adults  learn  a  lot  about  informal  "corporate"  life  and  their  jobs 

outside  of  the  formal  presentations  in  the  classrooms.  Instructors 
provide  some  of  this  informal  training  as  role  models  whose  incidental 
behavior  the  learners  observe  and  adopt.  Other  students  also 
significantly  affect  what  students  learn. 


Reading  List:  Hill.  H..  &  Sticht.  T.  (1980.  September).  Perspectives  on  battalion  training 

management  t Final  Report  for  VSAREUR  Field  Unit)  Alexandria.  VA:  Human 
Research  Organization. 

Kern.  R.  (1986)  Modeling  information  processing  in  the  context  of  job  training 
and  work  performance  In  T.  Sticht.  F.  Chang  &  S.  Wood  (Eds  ).  Cognitive 
science  and  human  resources  management  ( Advances  in  reading/language  research . 

.  Vol.  4)  Greenwich.  CT :  JA1  Press. 

Morgan.  M..  Hall.  D  T  .  &  Martier.  A  (1979).  Career  development  strategies  in 
industry:  Where  are  we  and  where  should  w 't  bc°  Personnel.  56(2).  1 3—3 1 
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44.  Monitoring 


Finding: 


Comment: 


Reading  List: 


and  Tailoring  an  Instructional  System 


Instruction  improves  when  managers  monitor  achievement  indicators, 
detects  when  the  value  of  any  indicator  moves  into  an  unacceptable 
range,  and  then  takes  focused  corrective  action  (tailoring). 


Monitoring  and  tailoring  of  instructional  systems  resembles  controlling 
physical  systems  such  as  heating  or  cooling  systems.  However,  the 
relevant  indicators  in  training  systems  are  less  precise  than  those  in 
physical  systems.  They  must  be  determined  by  examining  the  gt><ils 
of  the  schools,  the  management  practices,  and  objective  information 
about  students  and  instructors. 

Training  managers  can  monitor  direct  and  indirect  student 
performance  indicators  to  establish  priorities  for  improving  the 
system.  Direct  indicators  include  student  attrition  and  the  rate 
students  are  "set  back-'  to  repeat  lessons,  and  comprehensive  and 
performance  test  scores.  Indirect  indicators  include  student-instructor 
ratios  and  background  variables.  This  monitoring  requires  access  to 
longitudinal  records  and  considerable  information  processing.  With  a 
computer-based  information  system,  managers  can  identify  indicators 
with  values  that  are  in  an  unacceptable  range.  Over  time,  monitoring 
will  reveal  if  the  quality  of  instruction  is  being  improved 

Focused  corrective  action  or  tailoring  requires  a  deployable  resource 
to  respond  to  the  indicators.  For  example,  an  instructional  supervisor 
or  curriculum  standards  office  representative  might  visit  a  classroom 
or  school  to  confirm  (or  refute)  that  a  problem  exists,  diagnose  the 
situation,  and  propose  corrective  action. 

The  monitoring  and  tailoring  approach  assumes  that  fine  tuning  the 
instructional  system  can  improve  the  system  significantly.  The  system 
may  require  fundamental  changes  due  to  changes  in  technology  , 
resources,  or  society. 


Cooley.  W  W  ( 1 983.  June/July).  Improving  the  performance  of  an  educational 
system  Educational  Researcher.  4-12. 
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Cooks.  W  W  .  A  Lofinev  F  R  ( l^76)  Evaluation  research  in  edmtiuur.  Tutors 
pruu  ip'.t’s.  and  pra-.iut  New  York  John  Wiles  &  Sons 

Wjlhere.  H  J  ll*»N4.  Mas)  Improsing  the  produetmts  of  America  s  schoi.js 
Educational  Leadership .  -4 1 ( fs ) .  1  <4-3(1 


45.  Physical  Classroom  Environment 


Finding:  Classroom  light  level,  color,  temperature^umidity,  and  noise  level 

control  eiTects  student  perception,  attention,  and  achievement. 


Comments:  Research  indicates  that  in  a  well  designed  learning  environment  a 

greater  level  of  learning  occurs  than  in  a  poorly  designed  educational 
environment.  For  temperature  variables,  an  air  temperature  between 
6S-70  CF  at  30“  from  the  floor  for  elementary  students  engaged 
in  sedentary  activities,  such  as  reading  and  writing  is  a  low  level 
heating  goal  in  the  winter.  For  older  students,  a  temperature  range 
of  68-74  T  at  30"  from  the  floor  in  the  winter  is  healthful  and 
comfortable,  if  the  humidity  is  kept  between  30-60% .  The  maximum 
temperature  for  adult  learners  for  sedentary  tasks  is  85  CF  and  65  CF 
for  active  tasks. 

Concerning  noise  factors,  noise  levels  above  96dbA  impacted  error 
rates.  The  noise  effects  of  continuous  and  intermittent  pure  tone, 
noise  at  fairly  low  levels.  7{)dbA.  for  high  cognitive  loading  tasks 
results  in  a  significant  decrease  of  learner  performance.  Background 
noise  apparently  interferes  with  learning  or  concentration  in  some 
learners  more  than  with  others.  The  highest  level  of  background  noise 
in  a  learning  environment  should  be  45db.  with  30db  being  optimal. 

Light  levels  should  be  directly  related  to  the  viewing  difficulty  of  the 
learning  tasks.  Research  indicates  that  insufficient  illumination,  glare, 
reflectance,  shadows,  low  brightness  contrast,  and  flickering  affect 
human  performance  and  can  be  controlled  by  good  lighting  and  color 
decisions.  For  reading  tasks,  the  lighting  level  should  be  540-755 
lux:  for  bench  work  540-1000  lux;  and  for  learning  rooms  (ambient) 
500-750  lux.  Extreme  light  level  contrast  must  be  avoided,  as  there  is 
minimal  adaptation  required  for  moving  from  a  brightly  lit  area  to  a 
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darker  area.  Constant  adaptation  from  these  varying  degrees  cf  light 
can  cause  eyestrain  and  headaches.  Research  data  neither  consistent!) 
supports  claims  that  windowless  classrooms  will  allow  increased 
concentration  and  thus  higher  achievement,  nor  the  fear  that  the 
absence  of  windows  will  have  harmful  psychological  or  physical 
effects 

It  appears  that  color  directly  influences  individual  physiology 
as  measured  by  blood  pressure,  respiratory  rate  and  reaction 
time.  Bright,  light  and  warm  colors  have  been  shown  to  initiate 
outward  a.  m  ines  and  actions.  Cooler,  softer  colors  tend  to  foster 
a  withdrawal  behavior,  and  a  red  background  generates  ideas  and 
actions.  A  green  color  is  desirable  for  mediation  and  for  performing 
tasks.  In  designing  learning  environments  it  is  suggested  that  a  variety 
of  colors  be  used. 

Ct>Ior  seems  to  influence  student  learning,  attitudes,  and  behavior 
The  impact  of  color  on  an  individual  changes  with  age  and  with 
cultural  background. 

Color  seems  to  have  a  direct  relationship  on  an  individuals  sense  of 
time,  red  makes  most  people  overestimate  time,  while  greens  and 
blues  in  he  environment  cause  an  underestimation  of  time.  On  the 
other  hand,  bright  colors  (such  as  red)  tend  to  increase  an  individuals 
activity  level,  so  consideration  of  the  types  of  objectives  to  be  taught 
in  an  area  should  be  considered  in  choosing  primary  classroom  or 
study  area  colors.  Some  general  guidelines  for  choosing  colors: 

•  Classrooms  or  laboratories:  greens,  blue-greens,  gray,  beige. 

•  Gyms:  neutral  tones  or  cool  colors. 

•  Auditoriums:  green,  aqua,  peach. 

•  Entry  areas:  pink  (also  useful  in  prison  holding  tanks  to  quiet 
prisoners),  or  neutral  tones. 

Changes  in  classroom  colors  can  affect  learners.  Changing  the  focus 
area,  or  changing  individual  attitudes,  in  a  classroom  may  also  be 
done  by  painting  the  classroom  a  light  neutral  color  and  then  using 
flood-lights  to  change  the  setting. 

Lighting  seems  to  have  a  profound  biological  effect  on  humans.  Both 
the  quantity  and  quality  of  light  are  important.  Natural,  full  spectrum 
tubes  (as  opposed  to  traditional  fluorescent  tubes),  reduces  individual 
stress  and  aggression,  yet  seems  to  be  more  intellectually  stimulating. 
While  light  level  preferences  differ  among  individuals,  bright  lighting 
seems  to  increase  student  achievement  for  most  students,  but  students 
preferring  low  lighting  did  better  under  lower  light  conditions,  a 
learning  environment  apparently  should  not  be  uniformly  bright  and 
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students  should  be  allowed  to  sit  where  their  light  preferences  direct 
them.  Guidelines  for  various  types  of  activities  can  be  found  in  the 
references  provided  below  . 


Mans  experiments  have  been  conducted  studying  the  effects  of 
noise  on  sustained  attention.  The  figure  below  illustrates  the  effects 
of  noise  on  the  incidence  of  errors  over  time  in  a  serial  reaction 
task  lasting  40  min.  Precautions  to  eliminate  the  effects  of  acoustic 
cues  were  taken  which  included  the  use  of  a  silent  keyboard  and 
ear  defender  headphones.  In  both  loud  (90dbC  and  soft  60dbC) 
conditions  the  noise  is  switched  off  at  the  end  of  the  third  quarter. 
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46.  Managing  Informal  Learning 


Finding:  A  focus  on  the  management  of  learning  can  improve  incidence  and 

quality  of  informal  learning  in  training  environments. 


Comments:  Individuals  obtain  much  of  their  knowledge  and  skills  outside  of 

formal  school  settings.  Some  informal  learning  is  negative  such  as 
learning  poor  work  habits  from  the  examples  of  others.  Managers  can 
influence  informal  learning  of  students  by: 

•  Applying  environmental  designs  for  learning:  for  example  placing 
posters  with  critical  information  in  mess  halls,  hallways,  and  other 
places  where  sailors  spend  time.  Messages  should  be  designed  for 
ease  of  learning,  motivation,  and  creating  awareness; 

•  Promoting  learning  requirements  for  off-time  during  watch 
standing,  placing  learning  materials  in  job/duty  sites,  requiring 
reading  of  job  and  training  materials  when  on-duty  during  slack 
periods,  and  verifying  that  the  assignments  are  done: 

•  Encouraging  all  personnel  including  individual  sailors  to  think: 
dedicating  "read  and  think"  time  during  duty  hours  for  personnel 
to  think  about  what  they  do  and  how  to  do  it  better. 


Reading  List:  Hall.  D  T..  &  Fukami.  C.  V.  (1979)  Organization  design  and  adult  learning. 
Research  in  Organizational  Behavior.  2.  325-1 6'. 

Hill.  H  .  &  Sticht.  T.  (1980.  September)  Perspectives  on  battalion  training 
management  (Final  report  L’SAREUR  Field  Unit).  Alexandria.  VA  Human 
Resources  Research  Organization 

Kern.  R  (1986),  Modeling  information  processing  in  the  context  of  job  training  and 
work  performance.  In  T.  Sticht.  F.  Chang  &  S.  Wood  (Eds.).  Cognitive  science 
and  human  resources  management  (Advances  in  reading/ language  research.  Vol 
4).  Greenwich.  CT:  JAI  Press. 

Morgan.  M..  Hall.  D.  T..  &  Martier.  A.  (1979).  Career  development  strategies  in 
industry-.  Where  are  we  and  where  should  we  be?  Personnel.  56(2).  13-31 

Rogoff.  B..  &  Lave.  J  (1984).  Everyday  cognition:  Its  development  in  social  context. 
Cambridge.  MA  Harvard  University  Press. 
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47.  Student  Instructor  Ratio  Tradeoffs 


Finding:  Increasing  moderately  large  student-instructor  ratios  in  basic  courses 

has  little  effect  on  student  learning.  Enlarging  class  size  frees  existing 
instructors  for  alternative  laboratory  training,  tutoring,  or  counseling 
activities. 


Comments:  Small  student-instructor  ratios  tend  to  promote  frequent  interactions 

between  students,  instructors,  and  materials.  Students  in  small  classes 
have  more  interest  in  learning,  achieve  more,  have  a  somewhat 
better  self-image,  and  have  a  better  quality  of  interaction  between 
student  and  teacher  than  do  students  in  large  classes.  Teachers  in 
small  classes  may  have  higher  morale.  In  colleges,  where  classes  are 
fairly  large,  both  instructors  and  students  prefer  smaller  classes,  but 
larger  classes  do  not  affect  student  academic  achievement.  When 
class  size  is  more  than  about  a  dozen  or  so  students,  there  are  fewer 
opportunities  for  students  to  participate  in  discussions.  In  lecture 
presentations,  class  size  makes  hardly  any  difference  because  students 
are  already  passive  and  interactions  are  minimal  Therefore,  for 
basic,  "academic"  training  courses,  class  size  —  unless  below  10  or  so 
students  —  does  not  affect  student  learning  until  it  gets  large  enough 
to  prevent  students  from  seeing  or  hearing  the  instruction. 

Changing  instructor-student  ratio  enables  managers  to  manage  their 
instructor  resources  in  ways  that  can  improve  student  learning. 
Instructors  now  relieved  from  presenting  duplicate  or  repetitive 
courses  can  prepare  other  presentations,  interact  with  students 
individually  or  in  small  groups,  conduct  laboratory  exercises,  or 
evaluate  and  revise  existing  courses. 

Individualized  instruction  and  small  group  instruction  have  different 
characteristics  than  does  large  group  instruction.  Large  group 
instruction  normally  relies  on  an  instructor  to  present  organized 
materials;  usually  with  a  "lesson  plan"  or  with  “instructional  media." 
Many  studies  of  class  size  indicate  that  a  reduction  of  a  large  class  of 
30  to  15  or  20  would  have  little  or  no  impact  on  student  achievement. 
Conservatively  stated,  the  addition  of  a  student,  or  two.  or  three  to 
a  class  that  is  over,  perhaps.  7  students  in  size,  will  not  demonstrably 
affect  the  quality  of  a  class. 
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Reading  List: 


IndiMdual  student  feedback  and  individualized  student  instruction 
tvpically  are  limited  to  less  than  a  minute  per  classroom  session,  a 
minute  to  each  student  in  a  classroom  of  twenty  students  would 
require  21*  min 

Laboratory-based  instruction  does  seem  to  decrease  in  quality  as  class 
size  increases  as  the  instructors  role  in  most  laboratory  sessions  is  one 
of  performing  individual  student  assessment  and  guidance 

Small  group  and  individualized  instruction,  in  most  courses,  does 
result  hi  higher  student  achievement  and  affect  than  large  group 
instruction  Teacher  roles  m  rhc.c  groupings  allows  a  greater 
ranee  of  teaching  strategies,  better  classroom  management,  and 
permits  more  attention  to  student  elasswork  and  homework  The 
relativclv  high  instructor  costs  m  these  groupings  often  makes  them 
prohibitively  expensive  tor  the  m.  isurahlc but  slight  increase  in 
student  achievement 


(laden.  A  (  !‘>wu  (  i.iss  sj/e  ,mJ  stutifRi  js hits imen'  Research-based  police 
alternative  l.iim  atu-nj.'  / niiuatiun  and  I’.du  \  Anaims.  12(2).  21>-22' 
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anatssis  tor  decision  makint;  Arhncton  VA  Educational  Research  Service 

Sljwn  R  ilusu)  Achievement  elfecis  ol  substantial  reduction  in  class  sue  In  R 
Savin  l  td  I.  Si ii(»c>'  and  liassroum  organization  Hillsdale.  NJ  t.rlbaum 

Williams.  D  D  .  Cook.  P  t  .  Ouinn.  B  .  A  Jensen.  R  P  tlRS5i  Imversttv  class 
size  Is  smaller  better  ’  Research  in  Higher  Education.  23i3).  307- IS 
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Introduction 

There  are  mans  approaches  to  the  evaluation  process,  but  the  most  essential  component 
is  gaining  knowledge  from  the  end-users  themselves.  How  this  is  accomplished  can 
be  a  vanelv  of  ways,  such  as  usability  laboratories,  on-site  and  field  evaluation, 
observations,  and  questionnaires.  The  method  to  be  used  is  determined  by;  (1)  the 
resources  (e  g.,  money,  laboratory  space,  filming  equipment)  that  are  available.  (2) 
the  geographic  iocation  of  the  users,  and  (3)  the  availability  of  trained  evaluators. 
Each  approach  has  particular  strengths  and  provides  the  evaluation  team  or  individual 
with  information  in  which  to  analyze,  plan  and  redesign  and  redevelop  the  product 
or  s\stem.  As  defined  by  Sohm  and  Bertrand  (1988).  the  evaluation  process  is  a 
tool  which  managers,  trainers,  and  designers  can  use  to  emphasize  the  facilitative 
dimension  and  iterative  process  of  evaluation.  They  further  state  that  evaluation  is  a 
learning  and  action-oriented  management  tool.  Evaluation  is  a  process  for  determining 
as  systematically  and  objectively  as  possible,  the  relevance,  effectiveness,  and  impact 
of  activities  and  products  in  the  light  of  their  objectives  in  order  to  improve  both 
current  activities,  products,  systems,  and  provide  for  future  planning,  programming, 
and  decision-making  (Sohm  &  Bertrand.  1988). 


Purpose 

The  intent  of  this  paper  is  two-fold;  One,  to  describe  the  evaluation  process  of 
the  document  "What  Works"  and  second,  to  demonstrate  the  measured  value  and 
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Mcnuicance  ft  the  "What  Works''  document  The  evaluation  process  was  broken  into 
two  pha'-c'  Ph.i'c  one  occurred  one  y car  after  the  development  of  the  "Wh.it  Works" 
document,  and  Navy  personnel  were  the  surveyed  users  Phase  two  occurred  two 
years  after  the  What  Work'”  document  was  published  and  included  Navy  personnel. 
Iniversity  professors  and  Technical  College  instructors.  Each  evaluation  phase  used 
a  separately  developed  questionnaire  in  order  to  attain  pertinent  data  concerning  the 
value,  use.  significance,  and  future  of  the  "What  Works  "  document.  Both  questionnaires 
employed  quantitative  questions  as  well  as  qualitative  questions  which  required  written 
responses  and  comments  from  the  users, 

L'smg  both  qualitative  and  quantitative  questions  for  attaining  information  from  the 
users  provides  a  type  of  content  validity  for  the  surveys.  Thus,  with  these  types  of 
multiple  measures  the  reliabilitv  and  validity  of  the  responses  to  the  survey  can  be 
assumed  to  be  reasonably  high  The  objective  of  this  evaluation  process  was  to  gain 
knowledge  and  feedback  from  the  users  regarding  the  valuable  use  and  significance  of 
the  "What  Works"  document  This  evaluation  process  is  considered  to  be  a  formative 
evaluation  as  knowledge  that  was  gained  from  the  first  evaluation  procedure  w;  s 
incorporated  into  the  enhancement  and  development  of  the  newly  revised  second 
“What  Works"  document.  With  the  passage  of  time,  and  expanding  the  user  group  to 
non-military  settings,  allowed  the  developers  of  "What  Works"  to  further  incorporate 
and  integrate  new  suggestions  and  ideas  into  the  document 


Methodology 

Evaluation  Procedure  for  Phase  I 

The  original  distribution  of  the  "What  Works"  document  was  primarily  to  Navy 
personnel,  specifically  training  executives,  training  specialists,  and  instructors.  A  year 
after  the  initial  distribution,  a  questionnaire  was  developed  to  survey  these  users  about 
the  usefulness  and  likelihood  of  use  of  the  "What  Works"  document  The  developed 
questionnaire  was  mailed  to  these  users  and  a  response  rate  of  45%  was  attained  The 
total  sample  size  for  this  survey  was  40  training  executives.  37  training  specialists,  and 
39  instructors  for  a  total  of  1 16. 


Survey  Instrument  for  Phase  I 

The  first  evaluation  questionnaire  for  phase  I  was  comprised  of  seven  questions.  Three 
questions  were  a  Likert  four  point  scale  with  room  to  provide  additional  comments.  Four 
questions  were  open  ended  questions  requiring  written  responses. 

The  scoring  of  the  questionnaire  consisted  of  percentage  distributions  for  the  first  three 
questions.  For  the  open  ended  questions,  the  responses  were  content  analyzed  according 
to  the  appropriate  user  classification,  that  is  for  training  executives,  training  specialists, 
and  instructors.  Additionally ,  tables  were  developed  incorporating  the  trainers  responses 
regarding  the  index  that  was  provided  in  the  “What  Works"  document.  This  index  is 
provided  at  the  beginning  of  the  document  and  is  broken  into  the  three  occupational 
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categories  ot  training  execute  os.  training  specialists,  and  instructors  Cross-referencing 
the  U'cr-<  responses  with  this  index  was  conducted 


Evaluation  Procedure  for  Phase  11 

After  "What  Works'  was  developed  and  distributed  to  the  appropriate  Navy 
personnel,  mans  requests  were  received  from  other  trainers  of  adults  A  copy 
of  the  "What  Works"  was  sen'  to  these  individuals.  As  the  list  of  mailings  to 
universities,  technical  trade  colleges,  and  various  military  branches  geographically 
located  throughout  the  United  States  and  overseas  (Australia  and  the  United  Kingdom) 
enlarged,  it  was  decided  to  evaluate  again  the  effectiveness  of  “What  Works'"  on  this 
expanded  population.  The  total  sample  size  and  distribution  was  the  following:  Military 
personnel,  n  d  IS.  technical  colleges,  n  d  2.  universities,  n  d  3.  totaling  23. 


Survey  Instrument  for  Phase  II 

The  evaluation  questionnaire  for  phase  II  constituted  of  seventeen  questions  This 
questionnaire  focused  on  the  use  of  the  "What  Works"  document,  organization  and 
style  of  the  document,  value  and  significance  of  the  document  and  several  questions 
on  the  future  application  and  use  of  the  document.  There  were  fourteen  questions 
that  required  a  yes  no  response  with  room  for  comments.  Three  questions  that  were 
developed  are  on  a  five  point  Likert  scale  and  again,  there  was  room  for  responses 
from  the  users.  At  the  end  of  the  questionnaire  there  was  room  for  general  comments 
concerning  the  "What  Works”  document. 

The  scoring  of  the  ase  II  questionnaire  consisted  of  calculated  percentages  for  the 
yes/no  responses,  frequency  ratings  for  the  questions  concerning  the  organization,  the 
value  or  significance,  and  the  writing  or  readability  of  the  "W'hat  W’orks”  document.  For 
the  open  ended  responses  and  comments,  content  coding  was  employed  and  organized 
for  these  qualitative  measures 


Results 

Phase  l  Evaluation  of  "  What  Works ” 

For  determining  the  usefulness  of  the  “What  Works”  document  a  distribution  of  the 
reported  percentages  is  presented  in  Figure  1. 

This  figure  illustrates,  for  each  Navy  training  specialization — training  executive, 
training  specialist,  and  instructor — that  overall  the  “What  Works”  document  was 
regarded  as  very  useful.  For  the  training  executives,  more  than  40%  reported  th3t 
the  "What  Works”  document  was  useful.  Over  50%  of  the  training  specialists  indicated 
that  the  "What  Works"  document  was  useful,  and  this  result  was  consistent  for  the 
training  instructors.  Depicted  in  Table  1  are  the  ratings  from  all  three  training  users 
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Usefulness 

Figure  1.  Percent  of  usefulness  of  "What  Works"  for  all  three  training  groups  [phase  !  evaluation) 


Table  I 

Training  Evaluators  Range  of  Usefulness  for  Training  Instructors  as 
Indicated  b>  "W  hat  Works"  Instructor  Index  [Phase- 1  Evaluation] 


"What  Works  '  Number  of  evaluators  citing  index  items  for 
training  instructors  as  most  useful  or  least  useful  (question  1) 


Evaluators  range  of  usefulness 

Most  useful 

Least  useful 

Index  for  training  instructors 

*E 

•s 

•I 

•E  *S  *1 

[01  j  Instructor  classroom  role 

1 

1 

11 

1 

[02]  Instructor  classroom  leadership 

1 

10 

1 

[03]  Teaching  students  how  to  learn 

1 

13 

[04]  Promote  development  of  mental  models 

1 

3 

4 

[05]  Motivating  students 

1 

8 

[Obj  Cooperation  in  learning 

6 

v 

[07]  Student  control  of  learning 

5 

1  7 

[08]  Instructor  presentation  stimulates  learning 

9 

[09]  Managing  class  time 

1  1 

{10}  Practice 

3 

2 

[11]  Peer  teaching 

10 

16 

[12]  Out-of-class  assignments 

I 

8 

[13j  -sting  student  learning 

1 

4 

1 

[14]  Giving  feedback  to  students 

11 

[15]  Rating  instructors 

4 

All  of  the  above  topics  (01-15)  found  useful 

3 

12 

11 

*E  U  Training  executives  ( n  d  40) 

*S  d  Educational  specialists  (n  d  37). 
*1  d  Training  instructors  (n  d  39) 


1 
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concerning  the  usefulness  of  the  document  with  regards  to  the  "What  Works”  index 
that  i'  presented  in  the  front  of  the  document. 

Some  of  the  comments  that  were  gi\en  by  the  users  concerning  the  usefulness  include 
the  following,  "the  body  of  work  is  a  valuable  resource  and  reference  document."  and 
"it  is  a  highly  informative  and  enlightening  document." 

Figure  2  presents  the  percentage  distribution  for  the  likelihood  of  using  the  "What 
Works"  document. 


Likely  Not  likely 

Likelihood  of  use 

Figure  2.  Percent  of  likelihood  of  use  of  "W'hai  Works"  for  all  three  training  groups  (phase  ! 

evaluation) 


More  than  55%  of  training  specialists  and  instructors  rated  the  likelihood  of  use  very- 
high.  The  percentage  of  training  executives  rating  the  use  of  the  document  was  lower 
than  the  specialists  and  instructors,  by  approximately  35%.  The  open  ended  comments 
supported  the  high  use  of  the  document  for  the  specialists  and  instructors  as  many  of 
them  indicated  that  they  would  use  it  for  in-service  training,  as  a  reference  document, 
as  a  baseline  for  developing  training  curriculum  as  well  as  for  instructor  courses. 

Depicted  in  Figure  3  is  the  percentage  of  familiarity  of  the  material  that  was  presented 
in  the  “What  Works"  document. 

It  appears  that  the  material  was  mostly  familiar  to  the  training  executives  and 
specialists  as  opposed  to  the  training  instructors.  Shown  in  Table  2  are  the  evaluators 
grouped  comments  for  each  of  the  trainer  types  reflecting  a  more  detailed  expression 
of  how  the  document  "What  W’orks"  was  familiar  to  the  evaluators. 

Some  of  the  open  ended  comments  regarding  the  familiarity  of  “What  Works"  were  the 
following:  (1)  “May  be  familiar  with  “What  Works”  findings,  but  the  document  updated 
my  knowledge  and  provided  me  with  new  references  and  resources.”  (2)  “some  new 
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Mostly  tomiUar  Not  familiar 

Fomitiority 

Figure  Percent  of  familiarity  of  "What  Works"  topics  for  all  three  training  groups  [phase-1 

evaluation) 


Table  2 

Training  Evaluation  Comem  Coding  for  Open  Ended  Responses 
Regarding  Familiarity  of  "W’hat  Works"  Topics 
for  all  Three  Training  Groups  (Phase  -  1  Evaluation] 


""What  Works  ':  Number  of  evaluators  grouped  comments  for 
what  is  familiar  in  What  Works  (question  3) 

Trainer  types 

Evaluators  grouped  comments 

*E 

•s 

•I 

[01}  Yes  in  IT  school,  ves  familiar,  yes  instructor  basic 

7 

4 

20 

[02j  Most  basic,  common  sense 

4 

1 

j(i3j  Specifics  not  known,  some  new  ideas,  common  but  unwritten 

1 

1 

*> 

[04 1  OnK  aware,  if  not  school  grads  not  known,  expands  info 

3 

4 

jo?)  Very  new.  new  concepts,  new  terms,  specialist  index  new 

1 

3 

(06)  Implemented,  teach  now 

1 

1 

(07)  Lacks  implementation 

1 

j08j  Computer  info  is  new 

I 

1 

‘E  =  Training  executives  («  =  40). 

*S  =  Educational  specialists  (n  =  37). 
"I  =  Training  instructors  (n  -  39). 


concepts  were  presented,  especially  with  new  training  technologies.”  and  (3)  “aware  of 
the  basic  concepts,  but  an  applied  research  approach  was  presented.” 

Question  number  four  on  the  first  evaluation  questionnaire  was  an  open  ended 
question  requiring  a  written  response.  These  comments  were  subsequently  content  coded 
for  each  type  of  training  professional.  Table  3  depicts  the  "What  Works”  document  index 
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Table  3 

Training  Esaluators  Indications  of  Areas  that  mav  be  Applied  to  Improve 
Training  tor  "What  Works'"  Index  for  Training  Executives  [Phase  !  Evaluation! 


"  What  Works'"  Number  of  evaluators  citing  index  items  for  executives 
that  might  be  tried  and  applied  to  improve  training  (question  4) 


Trainer  type-- 


Index  for  (raining  executives 

*E 

•s 

*1 

[01]  School  learning  environment 

(02)  Managing  instructors 

2 

3 

[03 j  Evaluating  and  supervising  instructors 

3 

3 

1 

[04]  Managing  student  learning 

2 

2 

[05]  Monitoring  and  tailoring  an  instructional  svstem 

2 

o 

[Obj  Course  evaluation  and  revision 

2 

4 

[(.rj  lmitatinc  the  working  environment  for  learning 

1 

1 

3 

[OS]  Maintaining  skills  and  knowledge 

1 

1 

[09]  Siudents-insiructor  ratio  tradeoffs 

1 

1 

[]((]  Managing  informal  learning 

1 

1 

[11]  Planning  chances  in  conducting  training 

3 

1 

[12]  Cost  effectiveness 

1 

1 

[13]  Structuring  instruction 

2 

[14]  Computer-based  instruction 

2 

1 

(15]  Video  technologies  for  instructing 

5 

1 

1 

[16]  Training  devices  for  task  simulation  and  practice 

1 

[P]  Distributed  instruction 

I 

[IS]  Adopting  training  innovations 

1 

I 

All  of  the  above  topics  (01-18)  will  try  to  be  applied 

3 

3 

"E  =  Training  executives  (n  =  40) 

*S  =  Educational  specialists  (rt  =  37). 
"1  =  Training  instructors  |rr  =  39) 


specifically  for  training  executives  and  their  reflections  of  how  the  document  results  might 
be  applied  to  improve  training  considering  each  type  of  trainer  category,  i.e..  executives, 
educational  specialists,  and  instructors. 

A  few  of  the  more  frequently  mentioned  comments  include:  “planning  changes  in 
conducting  training."  “evaluating  and  supervising  instructors,”  “course  evaluation  and 
revision."  and  "video  technologies  for  instruction." 

Similar  to  question  four,  question  five  was  another  open  ended  response  item.  This 
question  specifically  asked  the  trainer  professional  to  list  any  ideas  on  how  the  “What 
Works"  document  may  be  used  in  a  training  environment.  The  most  frequent  responses 
were  organized  according  to  each  trainer  type  and  are  shown  in  Table  4. 

A  few  of  the  comments  included:  (1)  “Instructor  training  and  indoctrination,” 
(2)  “planning  changes  in  conducting  training,”  (3)  “evaluating  and  supervising 
instructors",  and  (4)  “course  evaluation  and  revision.” 

Table  5  shows  various  responses  for  each  training  professional  concerning  research 
summaries  that  could  be  useful  in  a  future  document  of  “What  Works.” 

The  most  frequent  responses  were:  (1)  Case  studies.  (2)  updating  and  including  new 
research.  (3)  examples  and  extensive  references.  (4)  cross-referencing  of  findings,  and 
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Table  4 

1  [.inline  Evaluators  Content  Coding  (or  Open  Ended  Responses  on  Hon  the 
-What  Works  '  mav  be  Used  for  all  Three  Training  Groups  (Phase  1  Evaluation! 


"What  Works  '  Number  of  evaUators  grouped  comments  that 
listed  ideas  on  how  the  document  ma>  be  used  (question  5) 

1  ramer 

T\  pc'" 

Evaluators  grouped  comments 

*E 

'S 

*i 

[01  j  IT  school 

4 

1 

5 

j(C|  Instructor  indoctrination.  1st  session,  training,  workshops,  handouts 

17 

8 

9 

(03J  Control  directives 

i 

[04]  Reference,  guide.  refresher 

6 

9 

i: 

(05 j  Inservice,  with  visuals,  req  d  reading.  GPI  school 

3 

3 

8 

[Oh]  Expand  topics 

1 

[<rj  Survev  of  schools 

1 

[OS]  New  siaff.  managers,  discussion  croups 

4 

6 

1 

[09]  CNET.  CNTECHTRA 

[10]  Instructor  evaluation,  monitor,  training,  feedback,  academic  skills 

1 

3 

N 

(11  j  Seminar 

1 

{ 1 2 j  Get  more  executives  involved 

1 

[)3]  Video 

1 

[14]  Design  of  instruction 

1 

i 

[15]  Semi-annual  performance  counseling 

i 

[16]  Distribution  implemented 

n 

i 

1 

"E  =  Training  executives  (n  =  40) 

*S  =  Educational  specialists  ( n  =  37). 
*!  =  Training  instructors  (n  =  39). 


(5)  broad  dissemination  of  "What  Works"  and  mandating  the  document  in  all  instructor 
training  courses. 


Phase  II  Evaluation  of  “ What  Works  ' 

Percentages  of  yes/no  responses  were  calculated  for  twelve  of  the  fixed  response 
questions  for  the  second  evaluation  questionnaire  of  "What  Works."  Table  6  presents 
these  percentages  for  the  appropriate  phase  II  evaluation  questionnaire. 

A  74%  yes  response  rate  was  reported  for  using  the  “What  Works”  document.  A 
variety  of  uses  were  indicated  by  the  training  professionals  and  users  as  provided  in  the 
comment  section.  Some  of  the  more  interesting  comments  include:  (1)  "Planning  for 
research  in  skill  retention.”  (2)  "understanding  perceptual  motor  skills  and  skill  decay,” 
(3)  "used  as  a  format  for  developing  videoteietraining".  (4)  “used  as  a  baseline,  starting 
point  to  develop  training  as  well  as  a  checklist  for  training.”  (5)  "use  of  tasks/functions 
for  managers/instructors  on  what  they  should  be  concerned  with.”  (6)  "used  in  graduate 
training,  instructions  courses  as  a  general  reference  document  for  my  graduate  and 
teaching  assistants."  and  (7)  "used  in  my  cost/benefit  analyses." 

The  frequency  rating  for  the  value  and  significance  of  the  “What  Works"  document 
is  illustrated  in  Figure  4. 


112 


NNhat  W  arks  in  Adult  Instruction 


4'V 


Table  5 

I  raining  Evaluator'  General  Comments  for  Other  Future  Research 
Summaries  for  all  Three  Training  Groups  |Phase-l  Evalua'ionj 


W  hat  Work'".  Number  of  evaluators  grouped  comments  on  listing 
other  potential  tuture  research  summaries  (question  6) 

T 

ramer 

types 

Evaluators  grouped  comments 

*E 

•s 

•i 

[(•1)  AN'  materials.  CNET  support.  IT  school,  new  schoolhouses.  tech  availability 

5 

1 

i 

[fC|  External  appraisal,  instructor  evaluation,  prerequisite  position  training 

3 

[03]  Examples,  more  info,  extensive  references.  U  S  Dept  ED  19Kb 

*> 

i 

(04)  ASV.AB  research,  research  on  specifics,  specific  info 

1 

T 

[05 j  Training  aids  lecture,  practice  examples,  instructor  index  1.2.3 

1 

S 

[Obj  Research  on  specific  rating  skills,  latest  research 

I 

) 

jo"|  Spacine  instruction,  keeping  standards  consistent 

1 

1 

(Osj  Latest  automated  technologv .  controlled  research  feedback  technologv 

*> 

[0*4]  Survev  of  schools,  research  on  application,  impact  studv  skills 

1 

1 

1 

fill]  More  fleet  update  after  graduation,  feedback  from  fleet 

]11|  Attrition  unahsis.  htsiorv .  demographics 

1 

1 

(12}  Case  studies,  case  studv  dev  lations  time 

1 

1 

j  13]  Research  on  retention,  retest,  skill  retention,  retention  termination 

1 

3 

3 

[  14]  Include  in  IT  basic 

1 

[15]  Multiple  choice  tests,  tests,  analyze,  evaluate 

1 

I 

jib]  References  from  AEG.  NPRDC.  ACR  follow-up  critiques 

1 

1 

[Pj  Conference  summaries  specific  problems 

1 

jlhj  CBI  versus  NON-CB1  job  performance 

1 

jldj  Librarv.  tech  manuals,  in  rate  pubs 

1 

(2lt[  Design  lab  sessions 

1 

*E  =  Training  executives  («  =  40). 

’S  =  Educational  specialists  (rt  =  37). 
‘I  =  Training  instructors  (//  -  39) 


20  r 


Excellent - Poor 

( n  *  23) 

Figure  4  Frequency  rating  for  value  or  significance  of  "What  Works"  [phase-2  evaluation). 
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Table  6 

Summarv  of  Percentage'  of  Questionnaire  for  Phase-2  Evaluation  of  "What  Works 


Findings  of  the  second  evaluation  questionnaire  of  '  What  Works 
percentage  that  agree  (yes)  vs  disagree  (no) 

Survev  questions  jphase-2  evaluation) 

%  Yes 

%  No 

V 

III 

Did  sou  receive  the  NPRDC  "What  Works"0 

91% 

09% 

->  “V 

(2) 

Have  vou  received  other  research  summaries  similar  to  What  Works0 

17% 

83% 

23 

Q) 

Did  vou  c:**  W  hat  Works 1 

74% 

26% 

2? 

(?) 

Did  vou  have  anv  problems  in  usinc  W’hat  Works? 

05% 

95% 

20 

(7) 

In  What  Works  document,  would  the  use  of  examples  be  helpful0 

91% 

09% 

(8) 

Would  directions  on  using  What  Works  be  helpful  ' 

36% 

64% 

T7 

(10a) 

Do  sou  have  anv  personal  experiences  (examples,  procedures  etc  ) 
tha.  could  be  included  or  added  to  the  document  of  What  W'orks0 

18% 

82% 

(10b) 

Could  we  use  sour  exampleis)  in  □  future  What  Works? 

80% 

20% 

5 

111) 

Are  there  additional  topics  that  vou  feel  should  be  added  to  What  W'orks0 

53% 

47% 

19 

(13) 

How  or  where  do  you  think  an  updated  version  of  What  W'orks 
should  be  published  ’ 

100% 

00% 

13 

(Ml 

W  ould  an  electronic  or  on-line  message  system  be  useful  in  conjunction 
with  What  Works? 

55% 

45% 

20 

(15i 

Would  having  a  help  line  or  a  specific  contact  person  for  assistance 
be  useful0 

70% 

30% 

20 

(161 

Would  a  hvperteM  or  similar  software  program  be  useful  for  locating 
research  and  disseminating  the  What  Works  conclusions0 

71% 

29% 

17 

V  =  Number  of  subjects  answering  the  question  (possible  n  =  23) 


As  shown.  18  training  users  rated  the  significance  of  the  document  as  very  good 
to  excellent.  Various  comments  were  provided  by  the  training  professionals  including 
statements  such  as  •‘excellent  piece  of  work  and  extremely  helpful."  “assisted  us  in 
developing  curriculums  for  the  real  world."  “nice  dissemination  of  applied  research 
and  examples  to  complement  the  theoretical  research.”  “useful  tool  and  nice 
cross-referencing  with  theoretical  and  applied  research."  and  “most  comprehensive 
review  I  have  seen  yet." 

A  very  low  percentage  was  reported  for  having  any  difficulty  or  problems  in  using 
“What  Works."  The  strength  of  the  organization  of  the  “What  Works"  document  is 
illustrated  in  Figure  5. 

Twenty-one  training  users  rated  the  organization  of  “What  Works"  as  very  good 
to  excellent.  One  user  commented  that  the  “What  Works”  document  was  a  “great 
dissemination  of  works  and  well  organized  considering  the  board  area  of  training  and 
learning  material  that  it  included.” 

The  frequency  ratings  concerning  the  writing  style,  readability  and  clarity  of  the  “What 
Works"  document  is  illustrated  in  Figure  6. 

Nineteen  training  professionals  rated  the  writing  style  and  readability  of  the  document 
as  very  good  to  excellent.  One  training  professional  user  wrote  “the  writing  style  of  the 
“What  Works"  document  was  refreshing  as  it  avoided  the  use  of  technical  and  nonsense 
jargon." 

All  of  the  evaluators  unanimously  (100%)  stated  that  publishing  an  updated  version 
of  “What  Works"  at  least  every  two  years  would  be  beneficial.  In  updating  the 
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(n  =  23) 

Future  5.  Frequency  rating  for  organization  of  "What  Works  '  [phase-2  evaluation) 


(n=23! 


Figure  6  Frequencs  rating  for  writing  style/readabiiin  clarity  of  "What  Works"  [phase-2  evaluation]. 


document  new  references  and  resources  could  be  added  as  well  as  keeping  training 
professionals  appraised  with  new  research,  trends,  and  solutions  in  the  training  field, 
especially  considering  applied  training  research.  Furthermore,  additional  topics  or 
expanding  current  topics  that  potentially  could  be  incorporated  into  future  “What 
Works-'  documents  were  provided  by  the  training  user  evaluators.  Some  of  the  topics 
elicited  include:  (1)  Team  training  and  evaluation.  (2)  new  research  on  instructional 
technologies.  (3)  evaluation  of  training  and  cost/benefit  analyses,  (4)  techniques  for 
developing  and  validating  test  materials,  (5)  approaches  for  effective  transfer  of 
training.  (6)  1  or  2  detailed  case  studies  of  how  to.  and  cross  referencing  to  the 
appropriate  material  in  “What  Works,"  and  (7)  bibliography  of  meta-analyses  and 
literature  reviews. 

* 


115 


M  M  ROBERTSON  a  at 


Questions  which  focused  on  the  future  application  and  use  of  "What  Works  ’  indicated 
some  interesting  percentages  (i.e..  questions  14,  15.  16).  One  result  was  the  issue  of 
prosidi"'’  an  on-line  message  system.  A  corresponding  55%  response  rate  was  reported 
bs  all  tvpes  of  training  professional.  Furthermore.  70%  of  the  respondents  indicated 
that  they  would  like  to  have  a  specific  contact  person  for  assistance  concerning  the 
application  and  use  of  the  findings  presented  in  the  "What  Works”  document.  Some  of 
the  comments  provided  by  the  training  evaluators  and  users  concerning  the  necessity  of 
havine  a  point  of  contact  were:  (1)  “Would  be  helpful  as  an  additional  resource/assistance 
to  the  document.”  (2)  “establish  a  continuous  contact  could  be  helpful  to  facilitate  the 
development  and  need  for  additional  topics.”  (3)  "helpful  as  it  would  assist  us  since  we 
don't  have  the  experience  and  resources'*,  and  (4)  “a  e-mail  system  could  possibly  extend 
the  access  to  the  “What  Works”  document  and  possibly  provide  for  on-line  help.” 

Regarding  the  presentation  format  of  "What  Works.”  over  70%  of  the  trainer  users 
reported  that  developing  a  hypertext  system  for  “What  Works”  would  be  useful  as 
cross-indexing  and  word  searching  would  be  more  efficient,  effective,  and  helpful. 
Additional  comments  provided  by  the  training  professionals  were  that  they  would 
like  to  have  a  hypertext  format  as  it  would  allow  for  the  ability  to  cross-reference 
and  index.  However,  one  limitation  to  this  presentation  format  was  noted  by  several 
training  users  involving  the  availability  and  cost  of  computer  systems  which  could  be 
prohibited  in  some  situations.  Also,  many  users  commented  that  with  the  hard  copy 
document  they  had  the  ability  to  pass  it  around  to  other  colleagues  and  could  study  it 
for  a  longer  period  of  time. 

Two  general  comments  that  were  provided  by  the  training  users  in  the  open  ended 
comment  section  regarding  the  "What  W'orks”  document  were:  (1 )  “The  document  was 
a  fine  contribution  to  the  education  and  training  field."  and  (2)  “I  found  the  document 
such  a  valuable  reference  and  resource  that  I  have  ordered  25  more  copies." 


Discussion  and  Concluding  Remarks 

This  two  phase  evaluation  process  has  been  successful  in  gaining  information  from 
the  end-user  who  is  the  training  professional  in  this  case.  Using  questionnaires  to 
survey  the  end-users  has  provided  the  developers  of  the  “What  Works"  document 
with  valuable  information  and  feedback  regarding  the  usefulness,  value,  organization, 
and  relevant  topic  selections  of  the  document.  With  this  type  of  formative  evaluation 
process,  further  development  and  enhancement  of  the  “What  Works”  was  accomplished 
and  its  effectiveness  was  further  measured  by  the  phase  II  evaluation  questionnaire. 
Moreover,  using  quantitative  and  qualitative  measures  in  assessing  the  effectiveness  of 
the  “What  Works"  document  provided  multiple  data  points  and  a  type  of  cross  content 
validation. 

It  does  appear  that  the  “What  Works"  document  is  useful  as  well  as  being  a  significant 
body  of  work  that  is  valued  by  a  varied  training  population.  Many  training  professionals 
reported  a  high  satisfaction  with  the  value  of  the  document  and  its  usefulness  in  planning, 
designing,  and  developing  instructional  curriculum  in  diverse  learning  environments. 
Furthermore,  the  organization  and  writing  style  was  well  received  and  rated  high  by 
a  significant  portion  of  the  training  users.  Even  though  there  was  a  high  percentage  of 
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reported  familiarity  with  the  material  presented  in  the  document,  many  users  further 
clarified  that  the  document  contained  new  references,  resources,  and  linkages  between 
theoretical  and  applied  research  in  the  instructional  and  training  settings. 

Future  presentation  formats  and  topics  W'ere  explored  and  it  appears  that  mans 
training  users  would  like  to  see  a  hypertext  format  as  it  would  provide  them  with  the 
capability  to  cross-reference  and  index  instructional  and  training  topics.  Also,  having 
the  capability  to  access  via  e-mail  could  provide  the  training  professional  with  needed 
assistance  and  potential  reference  sources  in  developing  and  evaluating  their  training 
curriculums.  Providing  an  updated  "What  Works”  document  every  two  years  was  noted 
by  many  training  users.  This  would  allow  for  incorporating  new  training  strategies  and 
research  findings,  especially  in  the  area  of  new  training  technologies.  Additional  topics 
could  also  be  added  and  incorporated  and  integrated  into  the  overall  organization  of 
the  topic  indexes.  Providing  new  and  relevant  case  studies,  possibly  from  some  of  the 
end-users  themselves,  with  cross-referencing  to  the  summaries  and  index  in  the  “What 
Works”  document  was  reported  frequently  by  the  training  professionals. 

This  two  phase  evaluation  process  provided  the  “What  Works"  developers  with  the 
necessary  information  in  order  to  further  enhance  and  expand  the  document  as  well  as 
demonstrating  the  measured  value  and  significance  of  the  “What  Work*  document. 
As  indicated  by  the  end-users,  the  training  professional,  this  document  is  a  worthwhile, 
valuable  resource  summary  of  educational  research  in  the  planning,  designing,  and 
developing  of  instructional  and  training  curriculums  for  the  adult  learner. 
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